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FOREWORD 


For  some  tine,  the  ARI  Fort  Knox  Field  Unit  has  been  engaged  in  the  de¬ 
velopment  and  application  of  innovative,  effective  techniques  designed  to  in¬ 
crease  the  quality  of  the  products  of  Army  Centers /Schools.  These  products 
Include  trained  soldiers  and  training  materials.  The  training  evaluation  and 
feedback  team  of  this  unit  performs  research  and  development  on  increasing  the 
quality  of  these  products  by  improving  the  information  flow  between  training 
developers  and  evaluators  and  units  in  the  field. 

To  Improve  the  quality  of  their  products,  training  developers  need  in¬ 
formation  about  the  products  from  users  in  the  field.  Presently  much  of  the 
information  available  to  training  developers  on  their  products  is  collected  in 
the  field  by  training  evaluators.  Training  developers,  however,  have  indicated 
that  the  feedback  that  they  receive  from  the  field  via  training  evaluators 
does  not  satisfy  all  of  their  feedback  needs. 

This  report  examines  the  types  of  feedback  presently  collected  by  training 
evaluators  and  developers  at  one  large  Center/School  and  Identifies  their  addi¬ 
tional  feedback  needs.  Present  feedback  sources  are  examined  to  determine 
their  accuracy  and  usefulness.  The  results  suggest  that  the  feedback  presently 
available  to  training  developers  is  lacking  in  both  specificity  and  objectivity 
and  thus  may  be  neither  accurate  nor  useful.  Recommendations  for  improving 
the  accuracy  and  usefulness  of  the  feedback  are  given,  which  have  implications 
for  TRADOC  and  other  Army  personnel  concerned  with  obtaining  accurate,  useful 
feedback  from  the  field. 
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FEEDBACK  NEEDS  OF  TRAINING  DEVELOPERS  AND  EVALUATORS 


EXECUTIVE  SUMMARY 


Requirement : 

In  order  to  improve  the  quality  of  their  products,  training  developers 
need  accurate,  detailed  feedback  from  the  field.  For  the  most  part,  training 
developers  depend  upon  Directorates  of  Evaluation  and  Standardization  (DOES) 
to  supply  the  feedback  needed.  Recently,  training  developers  have  indicated 
that  the  feedback  that  they  are  receiving  from  DOES  evaluators  is  not  meeting 
all  their  needs.  This  report  documents  the  kinds  of  feedback  presently  avail¬ 
able  to  training  developers  through  training  evaluators  or  other  sources,  and 
suggests  ways  to  improve  the  flow  of  useful  feedback  from  the  field  to  training 
evaluators  and  developers  at  Centers/Schools. 

Procedure: 

Thirty  division  and  branch  chiefs  from  the  Directorate  of  Training  De¬ 
velopments  (DTD)  and  the  DOES  at  a  large  Center /School  were  Interviewed  con¬ 
cerning  their  feedback  needs  using  an  interview  form  specially  developed  for 
the  purpose.  After  being  told  the  purpose  of  the  interview,  interviewees  were 
queried  about  the  types  of  feedback  currently  available  to  them.  For  each 
type  of  feedback  mentioned,  the  interviewees  were  asked  to  tell  where  they  get 
it,  how  they  use  it,  and  where  they  send  it.  Interviewees  were  then  asked  to 
state  any  needs  that  they  might  have  for  additional  feedback.  Results  of  the 
interviews  were  recorded  on  the  interview  form  by  the  authors.  Results  ob¬ 
tained  from  different  interviewees  were  integrated  for  drawing  conclusions  and 
making  recommendations. 

Findings: 

The  present  feedback  system  is  not  providing  all  the  feedback  that  train¬ 
ing  developers  need  for  Improving  training  programs  and  materials.  In  the 
case  of  training  materials,  more  feedback  is  needed  on  their  availability  and 
use.  In  the  case  of  training  programs,  considerable  feedback  is  currently 
available,  but  the  feedback  tends  to  lack  specificity  and  is  largely  derived 
from  subjective  sources.  To  increase  the  quantity  and  quality  of  feedback 
available  to  training  developers,  the  coordination  between  DTD  and  DOES  per¬ 
sonnel  should  be  increased,  DTD  personnel  should  increase  the  number  of  con¬ 
tacts  with  the  field,  more  emphasis  should  be  placed  on  hands-on  testing  and 
other  objective  data  collection  methods,  and  computer-based  data  analysis  and 
management  techniques  for  handling  feedback  should  be  developed. 

Utilization  of  Findings: 

The  findings  of  this  report  should  be  useful  to  any  group  or  agency  in¬ 
terested  in  obtaining  accurate  feedback  on  the  quality  of  their  products. 
Training  developers  and  evaluators  should  find  the  recommendations  included  in 
the  report  particularly  useful  for  improving  the  quality  of  the  feedback  that 
they  obtain. 
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SECTION  I 


INTRODUCTION 

In  order  to  develop  and  conduct  effective  training  in  an  efficient  manner 
the  Army  has  adopted  the  Instructional  Systems  Development  (ISD)  model.  The 
ISD  model,  as  described  in  US  Army  TRA.DOC  Pamphlet  350-30,  specifies  a  five 
phase  process  for  the  development,  conduct  and  evaluation  of  training.  In  the 
first  phase,  ANALYSIS,  the  job  is  analyzed  to  determine  what  tasks  should  be 
trained  under  what  conditions  to  what  standards.  The  second  phase  is  DESIGN, 
in  which  training  objectives  and  test  items  are  developed.  During  the  third 
phase,  DEVELOPMENT,  the  instructional  materials,  including  the  program  of 
instruction,  lesson  plans,  advance  sheets  and  training  aids,  are  created.  Dur 
ing  the  DEVELOPMENT  phase,  a  plan  is  also  developed  for  pilot  testing  or  vali¬ 
dating  the  newly  developed  materials.  Phase  four,  IMPLEMENTATION,  consists  of 
preparing  for  and  actually  delivering  the  instruction.  During  the  final  phase 
CONTROL,  the  quality  of  the  instruction  is  assessed  and  the  instruction  is 
revised  as  necessary. 

Much  of  the  responsibility  for  ensuring  that  training  development  at  the 
Center/School  proceeds  in  accordance  with  the  ISD  model  is  vested  in  the  Direc 
torate  of  Training  Developments  (DTD).  Some  responsibility  for  ISD,  however, 
is  reserved  for  the  Directorate  of  Evaluation  and  Standardization  (DOES)  and 
other  directorates  or  departments.  The  DTD  is  fully  responsible  for  the  ANALY 
SIS  phase  and  assumes  the  lion's  share  of  responsibility  for  the  DESIGN  and 
DEVELOPMENT  phases.  The  DTD  shares  responsibility  with  the  DOES  in  the  devel¬ 
opment  of  tests  and  validation  of  tests  and  instruction.  IMPLEMENTATION  is 
performed  by  the  various  commands  or  departments  responsible  for  particular 
areas  of  instruction  (e.g.,  the  maintenance  department  is  responsible  for 
maintenance  instruction).  In  the  CONTROL  phase,  the  DOES  is  responsible  for 
evaluating  the  quality  of  instruction,  identifying  problem  areas  and  referring 
the  problems  to  the  DTD  and  other  directorates/departments  (e.g..  Directorate 
of  Aviation,  Weapons  Department,  Maintenance  Department) ,  who  in  turn,  are 
responsible  for  making  the  necessary  course  revisions  and  putting  them  into 
effect . 

To  perform  the  various  phases  of  the  ISD  process,  the  DTD,  the  DOES,  and 
the  directorates/departments  need  feedback  from  units  in  the  field.  This  is 
especially  true  for  the  CONTROL  phase  in  which  the  quality  of  instruction  is 
assessed.  Instructional  quality  is  assessed  through  the  evaluation  of  the 
products  of  training  development  activities.  These  products  include  trained 
soldiers,  training  materials  (e.g.,  field  manuals,  technical  manuals,  ARTEPs) , 
and  training  programs.  The  DOES’s  mission  mandates  that  it  collect  data  to 
assess  training  products  and,  where  problems  are  identified,  provide  these 
data  as  feedback  to  the  appropriate  directorate/department  for  resolution. 

The  DTD  and  instructional  departments  need  to  be  informed  of  problems  with 
soldier  performance,  training  programs  and  training  materials  in  order  to 
correct  deficiencies  and  improve  their  products.  DTD's  charter  does  not  pro¬ 
vide  for  DTD  personnel  to  gather  data  for  feedback  purposes.  The  DTD,  there¬ 
fore,  must  depend  on  the  data  provided  by  the  DOES  for  the  feedback  it  needs. 
The  Instructional  departments  are  likewise  dependent  upon  the  DOES  for  their 
feedback  needs. 


There  is  a  certain  logic  in  having  training  evaluators  such  as  the  DOES 
assess  the  quality  of  the  products  of  training  developers  and  provide  feedback 
on  any  problems  identified,  rather  than  having  training  developers  evaluate 
their  own  products.  The  DOES  is  likely  to  be  more  objective  in  its  evaluation, 
and  it  is  more  efficient  for  a  single  agency  to  collect  the  feedback  for  sever¬ 
al  agenices  than  for  each  agency  to  collect  its  own.  Furthermore,  the  demands 
upon  the  time  of  the  units  from  whom  the  feedback  is  collected  are  probably 
less  when  a  single  agency  is  responsible  for  collecting  the  feedback.  On  the 
other  hand,  the  feedback  provided  to  the  DTD/ departments  by  the  DOES  may  not 
fulfill  the  needs  of  these  training  developers.  In  fact,  the  DTD  and  others 
have  recently  indicated  a  growing  concern  that  the  DOES  is  not  providing  the 
feedback  that  they  need.  They  suggest  that  much  of  the  feedback  that  they  re¬ 
ceive  from  the  DOES  is  too  general  and  is  not  useful  for  modifying  training 
materials  and  training  programs.  At  one  Center/ School,  training  developers 
have  asked  the  Army  Research  Institute  (ARI)  to  assist  them  in  identifying 
useful  field  performance  data  sources  and  developing  procedures  for  feeding 
field  performance  data  back  to  training  development  activities. 

ARI  to  date  has  completed  two  reports  aimed  at  improving  the  quality  of 
feedback  provided  to  Army  Schools/Centers.  The  first  (Burnside,  1981)  explores 
the  usefulness  of  existing  data  sources  (e.g.,  job  books,  SQT  results)  of  field 
performance  feedback,  including  general  recommendations  regarding  the  use  of 
different  feedback  methods.  The  other  report  (Burnside,  1982)  examines  the 
accuracy  of  the  most  commonly  used  method  for  obtaining  feedback  on  task  per¬ 
formance,  subjective  appraisal.  Feedback  obtained  by  this  method  consists 
largely  of  individuals'  judgements,  estimates,  or  opinions,  which  can  be  but 
are  not  always  externally  verified.  In  addition  to  these  reports,  ARI  is  devel¬ 
oping  and  testing  various  data  gathering  procedures  and  formats  for  collecting 
feedback  in  areas  where  the  feedback  needs  are  already  known. 

The  purpose  of  this  paper  is  to  identify  the  information  feedback  needs 
of  training  developers  and  evaluators  as  the  basis  for  developing  an  integrated 
feedbc..«.  system  for  feedback  of  information  from  centers,  schools  and  opera¬ 
tional  units  to  training  developers  and  evaluators.  Present  sources  of  feedback 
and  feedback  needs  are  identified  through  interviews  with  training  developers 
and  evaluators  at  one  Center/ School .  Since  the  results  are  based  primarily  upon 
a  study  of  a  single  Center/School,  generalization  to  other  Centers /Schools 
should  be  conducted  cautiously.  Most  of  the  conclusions  reached  in  this  paper 
are  general  ones  which  should  apply  to  other  Centers/Schools.  Because  they  are 
involved  in  all  phases  of  training  development,  DTD  personnel  provided  the  data 
on  the  feedback  needs  of  training  developers.  DTD  personnel  are  involved  in  the 
performance  of  the  following  training  development  functions:  front-end  analysis; 
development  of  individual  and  collective  training  materials,  devices,  and  pro¬ 
grams;  and  preparation  of  camera-ready  copies  of  and  distribution  of  institu¬ 
tional  and  extension  training  materials.  It  was  felt  that,  based  upon  their 
experiences,  DfTD  could  adequately  express  feedback  problems  and  needs  from  the 
training  developer's  perspective.  Therefore,  other  training  developers  such  as 
those  in  the  instructional  departments  were  not  interviewed.  The  DOES  as  the 
primary  evaluator  and  provider  of  feedback  at  the  Center/School  was  selected  for 
obtaining  data  on  the  feedback  sources  and  needs  of  training  evaluators.  In 
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addition  to  gathering  feedback  on  personnel,  logistics,  and  training  problems 
from  National  Guard,  Reserve,  and  active  units  in  the  field,  the  DOES  collects 
data  on  the  effectiveness  of  courses  conducted  at  the  Center/School .  Other 
functions  performed  by  the  DOES  include  providing  and  coordinating  assistance 
to  units  in  the  field  and  coordinating  modernization  and  deployment  of  new 
weapons  systems.  Although  these  functions  are  as  much  a  part  of  the  DOES's 
mission  as  evaluation,  this  report  will  emphasize  the  evaluative  function 
since  much  of  the  feedback  collected  by  the  DOES  is  collected  for  evaluation 
purposes. 

This  paper  concentrates  on  the  feedback  needs  as  they  apply  to  the  products 
of  TRADOC  Centers/Schools  to  include  trained  soldiers,  training  materials,  and 
to  a  limited  extent,  training  programs.  The  paper  is  comprised  of  five  sections. 
Following  this  section.  Section  II  describes  the  methods  that  were  used  to  iden¬ 
tify  the  feedback  needs  of  training  developers  (DTD's)  and  training  evaluators 
(DOES's).  Section  III  describes  the  collection  and  use  of  feedback  data  by 
evaluators  at  one  DOES  and  lists  feedback  needs  as  perceived  by  the  DOES  divis¬ 
ion  and  branch  chiefs.  Section  IV  describes  information  feedback  available  to  a 
representative  DTD  from  several  sources,  including  the  DOES,  and  tells  how  each 
type  of  feedback  is  used  by  the  DTD,  with  special  emphasis  on  DTD's  feedback 
needs.  Based  on  the  feedback  needs  as  stated  by  training  developers  and  evalua¬ 
tors  and  on  previous  analyses  of  feedback  sources  and  methods  (Burnside,  1981, 
1982),  the  final  section  presents  conclusions  about  the  feedback  methods  used  in 
gathering,  analyzing  and  distributing  feedback  information  on  Center/School 
training  development  and  administration  procedures. 

While  this  paper  deals  primarily  with  feedback  on  soldier  performance  and 
the  use  of  training  materials,  many  of  the  problems  and  issues  identified  apply 
equally  to  other  types  of  feedback  needed  by  various  agencies.  Other  types  of 
feedback  needed  might  include  feedback  on  the  utilization  of  training  resources, 
feedback  pertinent  to  the  assignment  and  retention  of  qualified  personnel,  and 
feedback  needed  for  managing  facilities  and  materiel.  The  recommendations  in¬ 
cluded  in  Section  V  may  be  used  by  any  group  or  agency  interested  in  obtaining 
accurate  continuing  information  on  the  quality  of  their  products.  Training 
developers  and  evaluators  should  find  the  recommendations  especially  helpful. 


SECTION  II 


INTERVIEW  METHODS 


Interviewees 


The  personnel  interviewed  were  division  and  branch  chiefs  from  the  DTD  and 
DOES  of  a  major  TRADOC  Center/School.  Each  division  or  branch  chief  interviewed 
was  responsible  for  management  of  a  major  functional  area,  as  outlined  below.  A 
total  of  30  division  and  branch  interviews  were  conducted  early  in  CY  1982, 
eight  in  the  DOES  and  22  in  the  DTD.  In  some  cases,  experienced  personnel  in  a 
branch  were  interviewed  jointly  with  or  in  addition  to  the  branch  chief,  partic¬ 
ularly  when  the  branch  chief  was  relatively  new  and  inexperienced  in  the  posi¬ 
tion.  In  the  DOES,  three  division  chiefs  and  five  branch  chiefs  were  inter¬ 
viewed,  and  in  the  DTD  six  division  chiefs  and  16  branch  chiefs  were  interviewed. 
These  personnel  represented  all  the  available  division  and  branch  chiefs  occupy¬ 
ing  permanent  operational  positions;  chiefs  of  special  teams  or  administrative 
offices  were  not  included  in  the  interview  sample.  Eighteen  of  the  30  division/ 
branch  chiefs  interviewed  were  military  officers,  and  the  remainder  were  civil¬ 
ian  personnel.  The  experience  of  the  interviewees  in  their  duty  positions  ranged 
from  a  few  months  to  seven  years.  While  no  two  DTD's  or  DOES's  are  organized 
exactly  the  same,  the  personnel  interviewed  are  representative  of  those  in  any 
TRADOC  Center/School  DTD  and  DOES.  The  functional  areas  addressed  in  the  DOES 
interview  sample  included  evaluation  of  training  at  the  Center/ School,  assess¬ 
ment  of  personnel,  logistics,  and  training  matters  in  active  Army,  Reserve,  and 
National  Guard  units,  and  systems  analysis.  In  the  DTD  the  functional  areas 
addressed  included  front-end  analysis,  development  of  individual  and  collective 
training  materials,  programs,  and  devices,  production  of  training  materials,  and 
distribution  and  management  of  extension  training  materials. 

Interview  Form 


In  order  to  insure  that  a  consistent  structure  was  followed  throughout  the 
interviews,  a  simple  form  was  developed  and  followed.  This  form  was  modified 
based  upon  experiences  during  early  interviews,  and  the  final  version,  consis¬ 
ting  of  three  worksheets,  is  at  Appendix  A.  The  first  worksheet  contains  an 
outline  of  the  introduction  to  the  interview  and  a  general  inquiry  concerning 
the  types  of  feedback  currently  available.  The  second  worksheet  contains  spe¬ 
cific  questions  about  the  feedback  identified  on  the  first  worksheet,  and  the 
final  worksheet  contains  a  general  inquiry  about  the  types  of  feedback  needed 
that  are  not  currently  available.  Blank  notebook  paper  was  used  to  record  any 
information  obtained  during  the  interviews  that  did  not  readily  fit  on  one  of 
the  worksheets. 

Interview  Procedures 


The  two  interviewers  for  this  project  were  the  authors  of  the  present  re¬ 
port.  Both  these  individuals  had  experience  conducting  similar  interviews  in 
previous  research  projects,  and  they  practiced  interview  administration  together 
to  insure  that  they  followed  the  same  interview  procedures.  Each  interview  was 
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conducted  in  the  interviewee's  work  area  or  a  convenient  conference  facility. 
The  interviewer  recorded  the  interviewee's  responses  on  the  appropriate  work¬ 
sheets.  An  introduction  to  the  purpose  and  procedures  of  each  interview  was 
given  following  the  outline  on  Worksheet  1.  Each  interviewee  then  listed  the 
major  types  of  feedback  available  to  him  or  her,  and  these  were  recorded  on 
this  worksheet.  He/she  then  answered  the  questions  on  Worksheet  2  for  each 
type  of  feedback  listed  on  Worksheet  1.  The  interviewer  completed  separate 
copies  of  Worksheet  2  for  each  type  of  feedback.  The  major  types  of  feedback 
needed  but  not  available  were  then  recorded  on  Worksheet  3,  along  with  any 
general  comments  received.  In  some  cases  a  return  visit  or  telephone  call 
was  made  to  an  interviewee  to  clarify  information  obtained.  Entries  on  inter¬ 
view  worksheets  were  revised  as  necessary  after  the  interview  to  insure  that 
they  contained  complete  and  readable  information. 
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SECTION  III 


DOES  FEEDBACK  SOURCES  AND  NEEDS 

As  one  of  their  missions,  DOES's  provide  feedback  to  Centers/Schools  con¬ 
cerning  problems  existing  at  training  institutions  and  in  the  field.  DOES's 
use  a  variety  of  methods  to  gather  the  necessary  information  and  disseminate 
it  to  the  appropriate  directorates/departments  at  the  Centers/Schools.  The 
methods  employe  1  by  DOES  in  collecting  and  disseminating  the  feedback  informa¬ 
tion  largely  determine  the  quantity  and  quality  of  feedback  available  to 
Centers/Schools . 

This  section  describes  the  kinds  of  feedback  provided  to  the  directorates/ 
departments  at  one  Center/ School  by  the  DOES  at  that  institution.  For  each 
type  of  feedback  listed,  methods  used  by  the  DOES  to  obtain  and  distribute  that 
information  will  be  identified  and  discussed.  As  a  matter  of  convenience. 
Information  obtained  from  units  in  the  field  and  information  collected  at  the 
institution  will  be  discussed  separately.  Following  the  discussion  of  types  of 
feedback  and  feedback  methods,  the  techniques  used  by  the  DOES  in  reducing  and 
analyzing  feedback  information  are  discussed.  This  is  followed  by  a  general 
discussion  of  the  usefulness  and  reliability  of  the  feedback  presently  provided 
by  the  DOES.  The  section  concludes  with  a  description  of  the  feedback  needs  of 
the  DOES. 

Although  the  data  presented  in  this  section  are  based  on  the  study  of  a 
single  DOES,  the  types  of  feedback  discussed  and  many  of  the  problems  associa¬ 
ted  with  the  collection  and  dissemination  of  this  information  are  common  to 
other  DOES's  and  thus  may  provide  valuable  lessons  for  DOES's  in  general.  This 
section  may  also  be  of  use  to  departments  and  other  directorates  for  under¬ 
standing  what  types  of  feedback  DOES's  obtain  and  how  they  obtain  them. 

Feedback  From  The  Fieli 


DOES's  generally  send  evaluation  teams  to  units  in  the  field  to  obtain 
information  about  problems  the  units  might  be  experiencing.  The  DOES  in  this 
study  sends  teams  consisting  of  two  to  five  members  to  each  active  and  reserve 
unit  once  every  two  or  three  years.  The  members  of  the  team  typically  include 
individuals  knowledgeable  in  the  areas  of  personnel,  logistics  and  training. 
During  each  visit  the  evaluation  team  administers  various  surveys,  question¬ 
naires,  and  informal  interviews  to  selected  unit  personnel.  Neither  systematic 
observation  nor  formal  testing  is  used  in  gathering  the  information.  Surveys 
administered  by  DOES  personnel  to  graduates  of  the  training  institution  and 
their  supervisors  require  the  graduates/supervisors  to  rate  the  graduate’s  per¬ 
formance  on  specific  tasks.  Questionnaires  in  the  areas  of  logistics,  personnel, 
and  training  are  either  sent  to  the  unit  just  prior  to  a  visit  or  are  adminis¬ 
tered  on  the  spot  by  DOES  personnel.  The  questionnaires  include  questions  on 
availability  and  quality  of  personnel,  equipment,  and  training  materials  and  are 
generally  completed  by  the  ranking  officers  in  the  unit.  The  interviews,  most  of 
which  are  also  conducted  with  these  officers,  address  many  of  the  same  topics  as 
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the  questionnaires,  but  include  additional  questions  on  training  materials  and 
soldier  performance  and  a  few  follow-up  questions  based  on  the  unit's  responses 
to  the  questionnaires. 

Ratings  of  graduate  performance.  For  each  graduate-producing  course  taught 
at  the  Center/ School,  graduate  supervisor's  surveys  are  completed  by  supervisors 
of  recent  graduates.  These  surveys  require  supervisors  to  rate  on  a  five-point 
scale  the  performance  of  each  graduate  relative  to  that  of  the  average  performer 
on  specific  tasks.  The  supervisor  is  also  required  to  rate,  on  a  five-point 
scale,  the  criticality  of  each  task  and  estimate  how  frequently  the  graduate 
being  evaluated  performs  the  task.  Similar  surveys,  referred  to  as  graduate 
surveys,  are  completed  by  the  graduates  themselves,  but  only  by  graduates  of  NCO 
and  officer  courses.  In  these  graduate  surveys,  the  graduates  are  required  to 
rate  their  own  performance  on  each  task,  estimate  their  frequency  of  performance, 
and  judge  the  criticality  of  the  tasks.  When  the  responses  to  these  surveys  in¬ 
dicate  that  the  graduate  performs  certain  tasks  poorly,  a  follow-up  interview  may 
be  conducted  to  determine  what  parts  of  the  tasks  the  graduates  can't  do  and  why 
they  can't  do  them.  Separate  follow-up  interviews  are  conducted  for  tasks  per¬ 
formed  marginally  and  tasks  performed  especially  well.  One  interviewee,  how¬ 
ever,  indicated  that  follow-up  interviews  of  survey  responses  were  seldom  con¬ 
ducted  in  recent  visits.  The  results  of  supervisor  and  graduate  surveys  are 
rarely  reported  by  the  DOES,  and  it  was  not  clear  how  or  if  these  results  were 
being  used. 

DOES  personnel  have  indicated  that  graduate  supervisor's  surveys  are  used 
in  lieu  of  hands-on  performance  testing  because  the  demands  of  the  latter  are 
extensive  in  terms  of  time  and  resources.  However,  Burnside  (1982)  has  suggested 
that  subjective  appraisals  of  performance  on  specific  tasks,  like  those  required 
by  the  DOES  surveys,  may  not  be  accurate,  and  should  always  be  checked  against 
objective  measures,  such  as  observations  or  tests  of  hands-on  performance,  to 
ensure  their  accuracy.  Discussion  with  DOES  personnel  indicates  that  no  accuracy 
checks  have  been  made  for  the  graduate  and  graduate  supervisor's  surveys. 

Responses  to  questionnaires.  The  evaluation  team  administers  three  differ¬ 
ent  questionnaires  during  field  visits:  (1)  a  personnel  questionnaire,  (2)  a 
logistics  questionnaire,  and  (3)  an  Army  Training  and  Evaluation  Program  (ARTEP) 
questionnaire.  The  personnel  questionnaire  is  generally  administered  directly  to 
the  battalion  commander  by  the  chief  of  the  personnel  division  or  his  representa¬ 
tive  during  the  visit.  Logistics  and  ARTEP  questionnaires,  on  the  other  hand  are 
mailed  to  the  unit  just  prior  to  the  evaluation  team's  visit  and  a  member  of  the 
team  collects  the  completed  questionnaire  shortly  after  arrival  in  the  unit.  The 
logistics  questionnaire  is  completed  by  the  battalion  logistics  officer,  battalion 
maintenance  officer,  and/or  executive  officer  while  the  ARTEP  questionnaire  is 
usually  completed  by  the  battalion  operations  and  training  officer.  The  personnel 
questionnaire  Includes  questions  on  the  numbers,  experience,  and  training  of  the 
personnel  in  the  unit.  The  logistics  questionnaire  includes  additional  questions 
on  the  number  of  personnel  in  the  unit  having  desired  educational  and  experiential 
qualifications.  Questions  regarding  the  condition  and  availability  of  equipment 
needed  by  the  unit  also  appear  on  the  logistics  questionnaire  as  do  questions 
concerning  the  ability  of  unit  personnel  to  demonstrate  proficiency  or  understanding 


In  general  functional  areas  (e.g.,  troubleshooting,  use  of  publications).  The 
ARTEP  questionnaire  is  used  to  evaluate  the  ARTEP  with  emphasis  on  the  ARTEP 
document.  Included  are  questions  designed  to  assess  the  unit's  opinions  about 
the  usefulness  of  the  ARTEP  and  to  determine  how  the  unit  uses  the  document. 

While  some  questions  on  the  personnel,  logistics,  and  ARTEP  questionnaires 
may  provide  objective,  quantifiable  data,  overall  the  questionnaires  do  not 
provide  the  data  in  enough  detail  to  satisfy  the  needs  of  training  developers. 

As  presently  stated,  many  of  the  questions  identify  general  problems  but  do  not 
pinpoint  the  source  of  those  problems  so  that  the  problems  can  be  eliminated. 

In  order  to  satisfy  the  feedback  requirements  of  training  developers,  some  of 
the  questions  Included  in  the  personnel,  logistics  and  ARTEP  questionnaires  may 
need  to  be  redesigned  and  additional  questions  developed. 

Informal  interviews.  Another  type  of  information  gathered  during  field 
visits  consists  of  data  from  informal  interviews  with  battalion  and  company 
commanders  and  their  staffs.  Generally  the  interview  covers  personnel,  train¬ 
ing,  and  logistics  topics.  The  usefulness  of  training  materials  (e.g.,  ARTEPs, 
Skill  Qualification  Tests  (SQTs) ,  Field  Manuals,  Technical  Manuals,  Commanders 
Guides)  are  typically  addressed  during  the  interview,  as  are  some  general  ques¬ 
tions  about  soldier  performance.  Informal  interviews  are  conducted  by  one  or 
more  members  of  the  evaluation  team.  At  least  two  members  of  the  team  indicated 
that  they  use  a  topical  outline  as  a  checklist  in  conducting  these  interviews. 
The  data  collected  during  the  interview  are  recorded  as  long-hand  notes  by  the 
interviewer. 

The  DOES  surveyed  in  this  study  relies  heavily  on  the  information  collected 
during  interviews  for  identifying  problems  experienced  by  units  In  the  field. 
While  unit  personnel  respond  favorably  to  interviews,  preferring  them  to  ques¬ 
tionnaires  (Burnside,  1981),  lack  of  structure  in  many  interview  situations  may 
result  in  a  failure  to  capture  much  important  data  and  hinder  comparison  of  data 
across  units.  The  interviews  may  need  to  be  structured  by  including  a  standard 
set  of  questions  that  are  asked  of  every  unit  visited.  As  the  reliance  on  the 
interview  as  one's  primary  source  of  data  increases,  the  importance  of  structur¬ 
ing  the  interview  also  increases. 

Trip  reports.  The  DOES  uses  the  information  gathered  during  visits  to  the 
field  as  input  to  trip  reports  written  following  each  visit.  These  reports  are 
usually  written  by  the  evaluation  team  leader.  Each  trip  report  consists  of  a 
series  of  statements  about  the  unit  visited.  Many  of  the  statements  identify 
problems  that  the  unit  is  experiencing.  In  the  trip  reports,  personnel,  logis¬ 
tical,  and  training  problems  are  generally  listed  separately.  Typical  problems 
listed  in  trip  reports  include  the  unavailability  of  certain  training  materials 
and  equipment  or  unit  dissatisfaction  with  the  materials  and  equipment  that  are 
available.  Among  other  problems  frequently  mentioned  are  shortcomings  in  insti¬ 
tutional  training  programs  or  perceived  deficiencies  in  the  performance  of  the 
graduates  of  those  programs.  Personnel  shortages  of  personnel  having  skills 
critical  to  the  functioning  of  the  unit  is  another  frequently  mentioned  problem. 
Although  any  problem  identified  during  a  field  visit  might  be  included  in  a 
trip  report,  one  team  leader  for  the  evaluation  team  stated  that  frequency  of 
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occurrence  and  problem  criticality  are  prime  considerations  in  deciding  which 
problems  to  include.  However,  no  running  tally  is  kept  on  the  frequency  of 
occurrence  of  different  problems,  and  there  are  no  set  criteria  for  judging 
either  the  frequency  or  criticality  of  a  problem.  Selection  of  problems  for 
inclusion  in  trip  reports  then  may  best  be  described  as  based  on  the  subjective 
judgment  of  the  team  leader. 

Trip  report  extracts.  Once  assembled,  the  complete  trip  report  with 
executive  summary  and  inclosures  is  provided  to  the  Commanding  General  of  the 
Center/ School  for  his  review.  The  Commanding  General  routinely  uses  information 
from  trip  reports  in  his  monthly  letter  to  commanders  in  the  field.  Upon 
approval  of  the  Commanding  General,  applicable  portions  of  trip  reports  are  ex¬ 
tracted  and  transmitted  to  the  Center/School  departments/directorates  and  to 
major  commands  (TRADOC,  FORSCOM,  DARCOM) .  To  maintain  the  anonymity  of  the 
units  evaluated,  unit  designations  are  normally  deleted  from  the  trip  report 
extracts.  The  trip  report  extracts  are  the  primary  means  by  which  the  DOES 
informs  the  Center/School  departments/directorates  and  major  commands  about 
problems  experienced  by  units  in  the  field.  Trip  report  extracts  usually  include 
a  request  that  the  DOES  be  informed  of  any  actions  that  the  department /directorate 
takes  to  address  the  problems  identified. 

Unit  training  materials.  Units  surveyed  during  field  visits  often  voluntar¬ 
ily  provide  training  materials  that  they  have  developed  for  their  own  use.  Upon 
their  return  to  the  Center/School,  DOES  evaluation  team  personnel  place  these 
materials  in  a  file.  If  the  materials  are  particularly  good,  or  if  they  are 
likely  to  be  of  general  interest,  they  may  be  added  to  the  trip  report  as 
inclosures. 

Other  field  feedback  media.  Besides  providing  input  for  trip  reports,  the 
information  obtained  from  the  surveys  and  questionnaires  completed  during  field 
visits  provides  feedback  in  other  ways.  For  example,  completed  ARTEP  question¬ 
naires  are  sent  to  the  DTD,  where  they  are  available  as  feedback  for  updating 
or  modifying  ARTEP  documentation.  The  DOES  personnel  division  compiles  quarter¬ 
ly  summaries  from  the  responses  to  the  personnel  questionnaire.  These  summaries 
present  totals,  averages,  or  other  descriptive  statistics  in  tabular  form  for 
each  questionnaire  item  across  the  units  surveyed  during  that  quarter.  Quarter¬ 
ly  summaries  are  kept  on  file  at  the  DOES  and  are  sent  to  the  Directorate  of 
Combat  Developments  (DCD) ,  the  Military  Personnel  Center,  or  the  Soldier  Support 
Center  on  occasion.  Responses  to  graduate  and  graduate  supervisor  surveys  are 
entered  quarterly  into  a  computer  via  a  remote  terminal  by  DOES  personnel  in  the 
systems  information  division.  DOES  personnel  have  programmed  the  computer  to 
compute  descriptive  statistics  and  generate  tables  of  these  data  for  making  sim¬ 
ple  comparisons .  These  data  are  used  by  the  training  division  of  the  DOES  in 
identifying  trends  in  the  abilities  of  graduates  to  perform  or  not  perform  spe¬ 
cific  tasks.  This  information  may  be  passed  along  to  the  DTD,  the  departments, 
or  the  Commanding  General  of  the  Center/School.  Data  from  the  logistics  question¬ 
naire  are  reviewed  by  the  Chief  of  the  Logistics  Division.  The  logistics  Chief 
extracts  the  information  pertaining  to  the  ability  of  unit  personnel  to  perform 
in  general  functional  areas  (e.g.,  troubleshooting,  use  of  publications)  from  the 
logistics  questionnaire,  sunanarizes  it,  and  forwards  the  results  to  the 
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Maintenance  Department.  The  Maintenance  Department,  however,  indicates  that 
the  information  provided  on  soldier  performance  is  too  general.  To  say  that 
soldiers  cannot  troubleshoot  is  not  sufficient.  The  Maintenance  Department 
needs  to  know  what  soldiers,  in  how  many  battalions,  cannot  perform  what  par¬ 
ticular  troubleshooting  functions  on  what  equipment  and  what  are  the  reasons 
for  the  poor  performance  (e.g.,  cannot  locate  faults,  inability  to  use  par¬ 
ticular  test  equipment) . 

Another  means  of  providing  feedback  on  unit  problems  and  needs  is  the 
Annual  Branch  Training  Team  (BTT)  Report.  The  author  of  the  BTT  report,  usu¬ 
ally  a  branch  chief  in  the  training  division  or  the  DOES  director,  selects  what 
he/she  feels  to  be  the  most  important  information  gathered  by  the  DOES  during 
field  visits  for  the  year  for  inclusion  in  an  Annual  BTT  report.  The  BTT 
Annual  Report  is  prepared  for  the  Commander  of  the  US  Army  Training  and  Doc¬ 
trine  Command  (TRADOC)  in  accordance  with  TRADOC  Circular  350-81-1.  DOES's  BTT 
report  is  not  distributed  to  other  departments/directorates  at  the  Center/ 
School,  although  the  final  BTT  report  produced  by  TRADOC  may  be  distributed  to 
the  Center/School  departments/directorates. 

The  DOES  collects  information  from  units  that  cannot  be  visited  during  a 
given  year  by  mailing  out  surveys  and  questionnaires  to  them.  Questionnaires 
and  surveys  distributed  to  units  by  mail  include  a  personnel  unit  assessment 
questionnaire,  a  logistics  unit  assessment  questionnaire,  graduate  supervisor's 
surveys  and  graduate  surveys.  Completed  questionnaires  or  surveys  are  returned 
to  the  DOES  through  the  mail.  DOES  personnel  estimated  the  return  rate  to  vary 
from  10%  for  the  graduate  surveys  to  40%  for  the  personnel  questionnaires.  The 
relatively  low  return  rate  for  the  graduate  surveys  might  be  due  in  part  to  its 
length,  which  varies  from  3  to  10  pages  for  different  graduates.  The  personnel 
questionnaire,  on  the  other  hand,  is  only  2  pages  in  length,  and  is  sent  to 
battalion  commanders,  who  may  be  more  willing  than  recent  graduates  to  complete 
questionnaires  of  whatever  length.  The  resulting  data  are  treated  essentially 
in  the  same  way  as  the  analogous  data  collected  during  field  visits,  except  that 
the  data  collected  through  the  mail  are  not  included  in  trip  reports. 

Although  distributing  surveys  and  questionnaires  through  the  mail  is  effi¬ 
cient  in  terms  of  time,  effort,  and  money,  and  provides  the  DOES  with  information 
that  might  not  otherwise  be  obtained,  there  are  serious  drawbacks  to  this  method 
of  collecting  feedback.  The  low  return  rate  makes  it  questionable  whether  the 
responses  are  representative  of  the  units  surveyed.  Another  drawback  is  that 
unit  personnel  often  perceive  mailed  surveys  as  chores  to  be  tolerated  and  do 
not  devote  as  much  thought  to  completing  them  as  they  could. 

Another  source  of  feedback  from  the  field  on  soldier  performance  is  the 
results  of  Comprehensive  Occupational  Data  Analysis  Program  (CODAP)  surveys. 

The  DOES  periodically  receives  print-outs  of  CODAP  results  from  the  Soldier 
Support  Center,  National  Capitol  Region  in  Alexandria,  VA.  CODAP  surveys 
typically  require  soldiers  to  rate  a  list  of  tasks  on  each  of  several  factors 
to  include  task  criticality,  how  frequently  the  task  is  performed,  and  how  well 
it  is  performed.  Several  DOES  personnel  mentioned  receiving  CODAP  results  but 
only  one  branch  chief  indicated  that  he  used  the  results.  That  branch  chief 


stated  that  he  extracted  data  from  the  CODAP  surveys  on  the  frequency  of  per¬ 
formance  of  tasks  and  used  the  data  in  conjunction  with  other  information  to 
determine  if  the  right  skills  are  being  emphasized  at  the  Center/ School .  He 
did  not  explain  just  how  he  combined  the  CODAP  data  with  other  information,  nor 
did  he  describe  how  he  decides  if  the  right  skills  are  receiving  emphasis. 

The  primary  source  of  data  for  matters  concerning  the  National  Guard  is 
the  National  Guard  Bureau.  The  Army  National  Guard  (ANG)  advisor  frequently 
receives  requests  for  information  about  the  National  Guard  from  various  agen¬ 
cies  (e.g.,  the  DTD,  the  DCD)  at  the  Center/School.  Typically  following  such  a 
request,  the  ANG  advisor  calls  the  National  Guard  Bureau,  which  either  provides 
the  needed  data  on  the  phone  or  sends  the  data  in  written  form  to  the  ANG  ad¬ 
visor.  The  ANG  advisor  then  relays  the  data  to  the  person  or  agency  requesting 
it.  Among  the  types  of  information  requested  are  SQT  results,  demographic  data 
on  ANG  personnel,  number  of  ANG  personnel  qualified  in  a  particular  MOS,  and 
questions  regarding  the  number  of  a  given  item  of  equipment  or  device  available 
in  ANG  units. 

Feedback  At  The  Center/School 


Much  of  the  information  feedback  gathered  on  soldier  performance  and  train¬ 
ing  materials  at  the  Center/School  is  collected  by  evaluation  personnel  in  the 
DOES  training  division.  Types  of  information  gathered  by  these  personnel  in- 
include  performance  data  on  objective  tests,  perceptions  of  graduate  performance, 
student  and  instructor  evaluations  of  courses  taught  at  the  Center /School,  and 
observations  of  the  training  process  used  in  conducting  the  courses. 

Course  test  results.  The  training  division  routinely  receives  results  of 
tests  administered  at  regular  intervals  from  the  test  and  evaluation  branch  of 
the  Directorate  of  Plans  and  Training  (DPT) .  These  results  are  received  in  the 
form  of  tables  displaying  the  number  of  soldiers  tested  and  the  number  receiving 
first-time  NO-GO's  on  each  task.  From  the  DPT,  the  DOES  also  receives  completed 
scoresheets  showing  each  soldier’s  performance  on  each  task.  These  scoresheets 
break  out  the  individual  soldier's  performance  by  subtask  and  give  reasons  for 
NO-GO's  on  each  task  and  subtask.  From  these  scoresheets  the  DOES  compiles  the 
percentage  of  soldiers  tested  who  failed  the  tasks  for  each  reason.  The  DOES 
also  uses  the  scoresheets  to  compile  the  number  of  tasks  failed  by  each  individ¬ 
ual  soldier.  The  results  of  DOES's  analysis  of  the  test  results  are  distributed 
to  the  DTD,  DPT,  and  the  various  directorates/departments  responsible  for  the 
instruct ion. 1 

When  the  testing  is  done  correctly  in  an  unbiased  manner,  the  test  results 
provided  by  the  DPT,  and  DOES's  analysis  of  them,  are  an  excellent  source  of 
objective  feedback  to  training  developers,  especially  when  coupled  with  observa¬ 
tional  data  on  the  training  preceding  them.  On  several  occasions  DOES  personnel 

^"Until  recently,  these  test  results  were  incorporated  in  semiannual  or  quar¬ 
terly  repot Cs  distributed  to  the  DTD  and  others.  But  the  present  branch  chief 
has  discontinued  these  reports  because  of  the  excessive  time  required  to  produce 
them. 
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have  observed  Che  testing  to  be  so  poorly  conducted  that  the  outcome  could  not 
be  trusted.  In  at  least  one  case,  DOES  personnel  administered  independent  tests 
to  a  sample  of  soldiers  following  a  course  because  of  low  confidence  in  the 
course  test  results.  The  results  of  this  independently  administered  test  were 
made  known  to  the  Commanding  General  of  the  Center/School,  but  were  not  dis¬ 
tributed  to  the  department  or  the  OTD. 

Skill  Qualification  Test  (SQT)  results.  Another  source  of  objective  infor¬ 
mation  received  by  the  DOES  training  division  is  SQT  results.  SQT  results  in 
the  form  of  a  computer  print-out  are  sent  to  the  DOES  from  the  Amy  Training 
Support  Center  (ATSC)  each  quarter.  The  SQT  results  are  broken  out  by  MOS  and 
skill  level.  The  print-out  lists  SQT  items  along  with  the  percentage  of  sol¬ 
diers  receiving  NO-GO's  on  each  item.  Items  having  high  failure  rates  (e.g., 

50%  or  greater)  and  the  tasks  with  which  they  are  associated  are  investigated 
by  the  DOES  to  determine  the  reasons  for  the  poor  performance.  The  DOES  checks 
the  wording  of  the  offending  item  and  looks  for  discrepancies  between  the  task 
as  tested  on  the  SQT  and  as  trained  in  the  unit  and  described  in  the  soldiers' 
manual.  All  SQT-related  problems  identified  by  the  DOES  are  referred  to  the 
DTD  for  its  consideration  and/or  resolution.  SQT  problems  may  also  be  brought 
to  the  attention  of  an  individual  test  specialist  working  for  ATSC  but  co-located 
with  the  DTD. 

Interviews  with  training  division  personnel  suggest  that  the  feedback  pro¬ 
vided  to  the  DTD  and  ATSC  on  SQT  problems  has  been  relatively  infrequent  in  the 
past  and  is  becoming  less  frequent  as  soldier  performance  on  the  SQT  improves. 
What  feedback  the  DOES  does  provide  to  the  DTD  or  ATSC  is  communicated  by  phone 
or  in  a  memorandum. 

Informal  feedback  seminars.  Training  division  personnel  conduct  informal 
feedback  seminars  with  officers  and  NCO's  within  the  first  two  weeks  of  the  NCO 
and  officer  advanced  courses  at  the  School.  Officers  and  NCO's  are  asked  to 
estimate  their  ability  or  the  ability  of  those  that  they  supervise  to  perform 
specific  tasks.  Questions  concerning  task  performance  are  asked  of  the  officers 
and  NCO's  as  a  group,  and  responses  are  recorded  as  longhand  notes.  Information 
obtained  through  these  feedback  seminars  is  used  in  conjunction  with  graduate 
and  graduate  supervisor  survey  responses  (collected  in  the  field)  for  identifying 
tasks  that  are  performed  poorly  by  graduates.  When  officers  indicate  that  a 
large  proportion  of  graduates  (precise  objective  criteria  were  not  stated)  can¬ 
not  perform  a  particular  task  satisfactorily,  training  division  personnel 
inspect  the  training  using  methods  described  in  the  following  paragraph  to  deter¬ 
mine  if  the  problem  lies  in  the  training  program.  Problems  identified  through 
the  use  of  graduate  surveys  and  feedback  seminars  are  typically  referred  to 
course  managers  of  the  appropriate  departments,  the  DTD,  or  to  the  Assistant 
Commandant  of  the  Center/School  for  action  or  information  purposes. 

Course  evaluations.  The  DOES  routinely  evaluates  the  courses  given  at  the 
Center /School.  Practically  all  new  or  pilot  courses  are  evaluated,  with  other 
established  courses  being  covered  to  a  lessee  degree.  Observations  are  collected 
during  training  and  testing  to  determine  the  effectiveness  and  efficiency  of  the 
courses.  Until  recently,  the  observations  of  training  and  testing  were  collected 
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routinely  using  structured  observation  forms  that  allow  the  observer  to  check 
for  the  occurrence  of  particular  training  events  and  record  his/her  comments. 
Using  a  training  evaluation  form,  the  evaluator  sought  to  determine  if  tasks 
were  demonstrated  and  practiced  to  standard,  if  the  lesson  plan  was  followed, 
and  if  needed  training  materials  and  job  aids  were  used.  Using  a  testing 
evaluation  form,  the  evaluator  observed  testing  to  determine  if  the  right 
tasks  were  being  tested  to  the  correct  standards  and  checked  for  bias  by 
examiners  in  administering  and  scoring  the  test.  The  TOES  branch  chief  respon¬ 
sible  for  course  evaluation  at  the  Center/School  indicated  that  these  forms  are 
no  longer  used  routinely  in  evaluating  training  and  testing,  and  that  he 
favors  allowing  a  subject  matter  expert  to  monitor  the  class  and  make  notes  on 
any  problems  observed.  Data  collected  during  course  evaluations  are  reviewed 
periodically,  looking  for  recurring  comments  or  something  that  is  totally  at 
variance  with  references  (e.g.,  soldier's  manuals,  field  manuals,  technical 
manuals),  lesson  plans,  or  expected  procedures.  When  problems  are  identified, 
they  are  relayed  to  the  instructional  departments,  the  DTD,  or  to  the  Deputy 
Assistant  Conmandant  for  Plans  and  Operations  for  corrective  action.  The  fre¬ 
quency  of  feedback  of  information  on  course  problems  to  the  instructional 
departments  varies.  For  new  courses  being  piloted,  feedback  may  be  given  on  a 
weekly  basis.  Feedback  on  other  courses  is  less  frequent  and  somewhat  irregular. 

Course  evaluations  are  an  important  part  of  an  overall  feedback  system. 

Tasks  that  are  not  performed  well  by  soldiers  in  the  field  often  are  not  trained 
well  at  the  Center/School.  DOES's  must  employ  reliable  methods  in  conducting 
course  evaluations.  Sending  a  subject  matter  expert  to  class  with  instructions 
to  record  any  problems  he/she  observes  does  not  consistently  produce  reliable 
data.  This  method  of  evaluating  training  is  likely  to  miss  the  most  critical 
problems  associated  with  training  and  testing  and  focus  on  minor  technical  dis¬ 
crepancies  in  the  course  content.  A  systematic  method  for  evaluating  training 
has  been  developed  by  ARI  and  is  documented  in  a  series  of  job  aids  (Kristiansen, 
1981;  Kristiansen  &  Witmer,  1982a;  Kristiansen  &  Witmer,  1982b;  Witmer,  1981). 

Student  questionnaires.  In  evaluating  School  courses,  the  training  division 
gathers  information  from  students  and  instructors.  Just  prior  to  graduation, 
student  questionnaires  are  administered  in  officer  and  NCO  courses.  Student 
questionnaires  provide  students  the  opportunity  to  rate  the  instructional  ade¬ 
quacy  of  the  training  they  received  for  each  task  taught  during  the  course.  When 
responses  to  the  student  questionnaire  show  that  20%  or  more  of  those  surveyed 
are  dissatisfied  with  the  training  given  for  any  one  task,  the  instructional  de¬ 
partment  responsible  for  the  training  that  task  and  the  DTD  are  informed  of  the 
potential  problem. 

The  DOES  bases  its  use  of  student  questionnaires  on  the  assumption  that 
officers  and  NCO's  are  qualified  to  judge  the  adequacy  of  the  training  that  they 
receive.  In  reality,  however,  students  may  not  be  qualified  to  judge  the  ade¬ 
quacy  of  the  training  they  receive.  It  is  unlikely  that  students  possess  the 
technical  sophistication  necessary  that  would  allow  them  to  evaluate  either 
course  content  or  the  training  and  testing  processes  employed.  Students'  eval¬ 
uations  may  also  be  unduly  influenced  by  the  personality  or  presentation  style 
of  the  instructor. 
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Instructor  questionnaires.  Periodically,  the  DOES  utilizes  instructor 
questionnaires  to  get  the  instructor's  perspective  on  the  adequacy  of  the 
course  design  and  the  quality  of  the  training  materials  provided.  Instructor 
questionnaires  are  most  frequently  used  in  conjunction  with  piloting  a  new 
course  or  during  a  special  study  of  a  particular  course;  they  are  not  used 
routinely  in  evaluating  training.  Instructor  questionnaires  are  either  mailed 
to  instructors  or  handed  directly  to  the  instructor  following  a  class.  In 
either  case,  the  instructor  is  asked  to  complete  the  questionnaire  independ¬ 
ently  and  return  it  directly  to  the  DOES.  Until  recently,  the  DOES  tabulated 
the  instructors'  responses  on  each  item  for  inclusion  in  quarterly  or  semi¬ 
annual  course  reports.  We  were  not  able  to  determine  how  the  results  from 
this  questionnaire  are  being  used  at  the  present  time. 

Training  inspections.  Co-located  with  but  not  directly  affiliated  with 
the  DOES  is  the  Commanding  General's  training  inspection  team.  The  primary 
function  of  the  team  is  to  periodically  inspect  the  training  of  every  unit  at 
the  Center/School.  This  includes  activities  responsible  for  institutional 
training  as  well  as  active  operational  units.  In  the  past  year,  the  training 
inspection  team  has  inspected  over  600  blocks  of  instruction. 

A  standard  training  inspection  report  data  collection  form  (Appendix  B) 
is  used  in  inspecting  the  training  and  reporting  the  results  of  the  inspection. 
The  data  collection  form  includes  items  to  be  observed  during  the  training  and 
testing.  The  form  has  a  place  for  recording  whether  or  not  the  event  described 
in  the  item  occurred  and  provides  some  space  for  inspector's  remarks  and 
recommendations.  The  space  provided  however  is  inadequate  and  does  not  encourage 
the  training  inspector  to  make  comments.  A  special  inspection  form  is  used  for 
inspecting  physical  readiness  training.  All  other  training  is  inspected  using 
the  standard  inspection  report  form. 

The  training  inspector  observes  the  training  and  testing  as  they  occur, 
recording  answers  to  specific  questions  relating  to  the  quality  of  the  training. 
When  an  answer  to  one  of  these  questions  indicates  a  problem  with  training  or 
testing,  the  inspector  writes  a  short  comment  beneath  the  item,  explaining  why 
the  question  was  answered  as  it  was.  Additional  comments  and  recommendations 
are  recorded  on  the  last  page  of  the  inspection  report.  As  a  last  step  in 
inspecting  the  training,  the  inspector  assesses  whether  or  not  the  training 
achieved  its  training  objective  and  records  his/her  assessment  in  the  report. 

This  assessment  is  to  a  certain  extent  subjective,  and  taken  alone  may  not 
adequately  reflect  the  effectiveness  of  the  training.  Following  the  inspection, 
the  inspector  briefs  the  Instructor  on  the  good  and  bad  points  of  the  training. 

Each  completed  inspection  report  is  reviewed  by  the  chief  of  the  Training 
Inspection  Team  for  completeness  and  internal  consistency.  The  training  inspec¬ 
tor  logs  the  class  in  on  a  large  chart  as  having  been  inspected,  which  lists 
the  unit  and  class  inspected,  the  instructor,  the  inspector’s  overall  assessment 
of  the  class,  and  the  date  of  the  inspection.  The  completed  report  is  then  filed 
by  date  for  ease  of  reference. 
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Copies  of  the  inspection  report  are  sent  to  the  Commanding  General  of  the 
Center/ School  and  to  the  unit  responsible  for  the  training  that  was  inspected. 
Depending  on  the  nature  of  the  problems  identified,  copies  of  the  report  may 
also  he  sent  to  the  DPT,  the  DTD,  or  the  School.  This  information  may  be  used 
by  the  units  to  modify  their  training  and  by  the  DPT,  the  DTD  or  other  School 
directorates/departments  to  change  factors  adversely  impacting  on  the  training. 
Some  units  also  utilize  the  information  provided  by  the  reports  in  keeping  track 
of  the  state  of  training  in  their  unit. 

Professionalism  surveys.  The  personnel  division  administers  professionalism 
surveys  to  officers  and  NCO’s  attending  the  officer  and  enlisted  advanced  courses 
at  the  School.  The  surveys  are  designed  to  gather  the  opinions  of  officers  and 
NCO's  in  several  different  areas,  including  the  professional  qualifications  of 
other  officers,  career  perceptions,  career  intentions,  career  satisfaction,  unit 
training  considerations,  and  the  perceived  quality  of  new  recruits.  Demographic 
data  are  also  collected  for  each  soldier  responding  to  the  survey.  The  responses 
to  these  surveys  are  analyzed  item-by-item  in  order  to  identify  trends  in  the 
data  suggestive  of  personnel  problems.  Descriptive  statistics  for  each  survey 
item  are  computed  periodically  and  a  roll-up  of  the  summarized  results  is  pro¬ 
duced  in  tabular  form.  The  descriptive  statistics  are  used  by  the  personnel 
division  to  identify  problem  areas.  Problems  thus  identified  are  referred  to 
the  directorate /department  who  is  in  the  best  position  to  do  something  about  them 
(e.g..  Directorate  of  Combat  Developments,  Soldier  Support  Center,  MilPerCen) . 

If  the  directorate/ department  is  unwilling  or  unable  to  resolve  the  problem,  the 
Deputy  Commanding  General  may  be  advised.  Problems  affecting  the  entire  command 
are  brought  to  the  attention  of  the  Center/School  Commanding  General.  If  the 
Commanding  General  agrees  that  the  problem  should  be  addressed,  then  he  directs 
the  appropriate  directorate/department  to  take  action  to  solve  the  problem. 


Data  Reduction  and  Analysis 


Hie  large  amount  of  data  collected  by  the  DOES  necessitates  that  the  data 
be  reduced  prior  to  distribution  to  other  agencies.  The  primary  strategy  used 
by  the  DOES  in  reducing  the  data  consists  of  summarizing  the  data  over  one  or 
more  variables  and  tabulating  or  cross  tabulating  the  results.  The  tabularized 
results  are  typically  expressed  as  a  frequency  count  or  as  a  percentage  of  the 
total  responses.  For  example,  course  test  results  may  be  expressed  in  terms  of 
the  percentage  of  soldiers  tested  who  passed  the  test  for  each  task,  and  the 
percentage  who  failed  for  each  of  several  reasons  (e.g.,  faulty  procedure,  too 
much  time) .  The  tabulated  results  are  either  forwarded  to  the  appropriate 
agency,  allowing  the  agency  to  draw  their  own  conclusions,  or  else  the  DOES 
identifies  problems  suggested  by  the  results,  and  communicates  their  interpre¬ 
tation  of  the  problems  to  the  agency.  In  some  cases,  the  DOES  uses  a  pre- 
established  criterion  as  a  means  of  identifying  significant  problems  (e.g.,  if 
20%  or  more  of  the  responses  to  an  item  suggest  that  a  problem  exists,  then  the 
DOES  identifies  the  problem  to  the  appropriate  agency  for  action).  In  other 
cases  the  DOES  looks  for  data  suggesting  the  existence  of  an  undesirable  trend 
or  information  indicating  a  problem  that  is  inherently  critical.  r  example, 
the  DOES  is  likely  to  identify  a  problem  for  further  action  if  it  affects  all 
or  many  units  served  by  the  Center/School,  or  appears  to  be  increasing  in 
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severity  over  time.  With  regard  to  trends,  nearly  one-half  of  the  division  and 
branch  chiefs  interviewed  stated  that  they  looked  at  trends  in  the  data  as  a 
means  of  identifying  problems.  None  of  these  chiefs  were  able  to  specify  pre¬ 
cisely  how  they  determine  the  existence  of  a  trend.  One  of  those  interviewed 
admitted  that,  at  least  for  him,  identifying  trends  was  done  subjectively  based 
on  his  best  judgment.  Other  remarks  made  in  the  course  of  the  interviews  sug¬ 
gests  that  the  DOES  does  not  have  clear  guidelines  that  they  consistently  use 
in  identifying  problems.  For  example,  several  interviewees  expressed  diffi¬ 
culty  in  determining  how  frequently  a  problem  had  to  be  mentioned  before  it 
could  be  considered  serious  enough  to  warrant  further  action. 

The  DOES  appears  to  use  statistical  procedures  for  data  reduction  on  a 
very  limited  basis.  The  use  of  statistical  analysis  for  arriving  at  conclusions 
and  making  comparisons  seems  to  be  the  exception  rather  than  the  rule.  During 
the  interviews  only  one  instance  was  mentioned  in  which  statistical  analysis 
was  used  by  DOES  personnel  to  draw  conclusions  regarding  the  data.  Although 
DOES  personnel  frequently  compare  results  across  units,  statistical  procedures 
are  seldom  used  in  making  the  comparisons.  It  appears  that  comparisons  are 
most  often  made  by  examining  the  data  in  the  light  of  one's  own  experience  and 
deciding  (subjectively)  whether  the  differences  or  discrepancies  are  of  suffi¬ 
cient  magnitude  to  constitute  a  problem.  This  subjective  assessment  of  results, 
which  varies  from  one  person  to  the  next,  is  a  poor  substitute  for  statistical 
analysis  or  other  objective  decision-making  procedures,  and  is  generally  not 
accepted  by  systems  analysts  or  other  members  of  the  analytic  community. 


Usefulness  And  Reliability  Of  Feedback 


From  the  previous  discussion  of  the  feedback  methods  employed  by  the  DOES, 
it  is  clear  that  the  usefulness  and  reliability  of  the  feedback  provided  to  the 
Center/School  is  not  all  that  it  could  be.  Estimates  of  graduate  performance  by 
graduates  and  their  supervisors  may  not  be  accurate.  Much  the  same  applies  to 
student  questionnaires.  The  questions  asked  on  questionnaires  and  during  inter¬ 
views  are  often  much  too  general  with  little  consideration  of  the  actual  feed¬ 
back  needs  of  directorates/departments  at  the  Center /School .  Interviews  con¬ 
ducted  by  the  DOES  lack  the  structure  that  is  necessary  to  be  a  primary  feedback 
source,  and  do  not  adequately  cover  battalion  personnel  at  levels  lower  than  the 
company  level.  Course  evaluations  in  which  observations  are  not  structured  and 
which  depend  on  a  subject  matter  expert  to  merely  record  what  he /she  sees  are  not 
likely  to  provide  useful  feedback  to  training  developers.  Subject  matter  ex¬ 
perts  often  become  so  involved  in  the  subject  matter  being  taught  that  they  all 
but  iffiore  the  manner  in  which  the  training  is  conducted.  Such  problems  tend 
to  reduce  the  usefulness  of  the  DOES's  feedback  to  the  Center/ Schools.  The 
usefulness  of  the  feedback  is  further  diminished  by  DOES's  lack  of  standardized 
objective  procedures  for  analyzing  their  data,  identifying  trends,  and  drawing 
conclusions  or  making  decisions  on  the  basis  of  their  data. 


Despite  the  problems  mentioned  above  most  of  the  DOES  branch  and  division 
chiefs  expressed  confidence  in  the  data  they  collected.  When  questioned  about 
the  reliability  of  the  information  feedback  that  they  get,  branch  and  division 
chiefs  indicated  that  they  believed  most  of  the  feedback  to  he  reliable. 
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Course  test  results  and  SQT  results  were  perceived  as  being  reliable  as  were 
questionnaire  and  interview  data  collected  during  field  visits.  Only  two 
sources  of  data  were  questioned  with  regard  to  their  reliability  or  usefulness 
One  branch  chief  questioned  the  reliability  of  the  graduate  supervisor’s  sur¬ 
veys  on  the  basis  that  the  results  may  be  affected  by  how  well  the  supervisors 
expect  the  graduates  to  do.  Another  branch  chief  expressed  confidence  in  the 
reliability  of  the  graduate  supervisor's  surveys  but  lacked  confidence  in  the 
ability  of  graduates  to  estimate  their  own  performance  on  specific  tasks  as  is 
required  by  the  graduate  surveys.  Professionalism  surveys  were  considered  to 
be  of  limited  value  as  a  means  for  identifying  problems. 

Feedback  Needs 


T  .-^dback  from  users.  Host  of  the  information  collected  by  the  DOES  is 
used  tor  identifying  problems  associated  with  processes  and  products  of  the 
Center /School.  As  mentioned  earlier,  the  DOES  typically  refers  these  problems 
to  the  directorate/depar tment  who  is  in  the  best  position  to  solve  them.  Prob¬ 
lems  are  usually  described  in  narrative  or  statement  form  in  a  report  or  mem¬ 
orandum  that  also  contains  information  briefly  explaining  the  source  of  the 
data.  Normally  the  DOES  does  not  make  recommendations  on  what  actions  should 
be  taken  to  address  the  problems.  The  solution  to  the  problem  is  left  up  to 
the  responsible  directorate/department,  as  is  the  decision  of  what  action  to 
take,  if  any.  In  one  sense,  the  DOES  can  be  said  to  turn  over  whatever  prob¬ 
lems  it  finds  to  the  responsible  directorate/department.  The  directorate/ 
department,  in  turn,  may  or  may  not  address  the  problem  and  is  not  obligated 
to  take  any  action  whatsoever  to  eliminate  the  problem. 

Generally  directorates/departments  receiving  information  about  problems 
from  the  DOES  do  not  inform  the  DOES  about  what  steps,  if  any,  they  are  taking 
to  eliminate  the  problems  identified.  Thus  the  DOES  receives  very  little  feed¬ 
back  from  users  on  how  the  information  that  it  provides  is  being  used,  or  even 
if  the  information  is  being  used.  Occasionally  a  user  will  send  a  reply  to 
the  DOES  indicating  their  agreement  or  disagreement  with  the  DOES's  conclusions 
and  describing  what  they  are  doing  to  address  the  problems.  One  branch  chief 
indicated  that  whenever  he  reported  problems  to  a  directorate /department,  he 
requested  that  the  directorate/depar tment,  by  a  specified  date,  inform  him  of 
what  actions  were  being  taken  to  address  the  problems.  Other  branch  chiefs 
stated  that  their  primary  means  of  determining  whether  changes  had  been  made 
in  response  to  the  problems  they  had  identified  was  to  make  additional  observa¬ 
tions  in  the  field  or  at  the  Center/School.  For  example,  if  the  same  problems 
were  found  during  an  inspection  of  a  particular  block  of  instruction  that  were 
identified  during  a  previous  inspection  of  that  instruction,  then  the  DOES  can 
be  pretty  sure  that  no  action  was  taken  to  eliminate  the  problems.  In  such 
cases  the  DOES  may  inform  the  Commanding  General's  office  that  no  corrective 
action  has  been  taken  to  eliminate  the  problems. 

The  lack  of  direct  feedback  from  directorates/departments  regarding  how 
they  use  (or  if  they  are  able  to  use)  the  information  provided  by  the  DOES  is 
a  major  impediment  to  obtaining  better  and  more  useful  feedback.  Without 
regular  user  feedback,  the  DOES  may  assume  the  feedback  they  provide  is  entirely 
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satisfactory  and  is  useful  in  correcting  Center/School  problems.  Statements  by 
DOES  personnel  expressing  confidence  in  the  reliability  of  the  feedback  they 
provide,  and  their  reticence  in  identifying  additonal  feedback  needs  indicate 
their  overall  satisfaction  with  the  feedback  they  provide.  User  feedback  is 
needed  to  inform  the  DOES  of  which  types  of  feedback  are  most  useful  and  to 
identify  additional  feedback  needs. 

Additional  feedback  needs.  DOES  division  and  branch  chiefs  indicated  few 
areas  in  which  they  needed  additional  feedback.  Two  of  those  interviewed  in¬ 
dicated  that  they  already  were  getting  more  feedback  than  they  could  handle. 

A  third  expressed  the  opinion  that  present  levels  of  data  collection  activi¬ 
ties  were  placing  excessive  demands  on  units  in  the  field. 

Of  those  expressing  the  need  for  additional  feedback,  three  wanted  more 
task-specific  feedback  regarding  hands-on  performance  in  the  field.  One  sug¬ 
gested  that  this  feedback  should  come  from  hands-on  tests  administered  to  sol¬ 
diers  in  the  field,  but  another  felt  that  DOES's  did  not  have  time  during  field 
visits  to  conduct  hands-on  tests.  One  of  those  wanting  more  task-specific 
feedback  was  especially  interested  in  getting  more  and  better  feedback  on  gun¬ 
nery  performance.  He  perceived  the  need  to  identify  individuals  who  contin¬ 
ually  fail  to  qualify  on  various  gunnery  tables  and  exercises  and  what  types 
of  engagements  caused  soldiers  the  most  difficulty.  He  suggested  that  each 
time  gunnery  is  conducted  the  following  kinds  of  information  should  be  recorded 
and  made  available  for  analysis:  (1)  type  of  gunnery  exercise  or  table;  (2) 
social  security  number  of  the  soldier(s)  participating;  (3)  when  and  in  what 
location  the  gunnery  occurred;  and  (4)  what  engagements  and  parts  of  engagements 
were  failed  by  each  soldier.  Three  interviewees  indicated  that  they  need  more 
information  on  the  availability  and  frequency  of  use  of  training  materials, 
including  technical  lessons,  devices,  and  publications.  Another  need  alluded  to 
was  the  difficulty  in  obtaining  demographic  data  on  individuals,  particularly  in 
the  reserve  component. 

Summary 

DOES's  provide  much  of  the  feedback  available  to  Centers/Schools.  Various 
methods  are  employed  by  DOES's  to  provide  this  feedback  and  distribute  it  to 
the  appropriate  directorate/department.  Some  of  these  methods,  however,  produce 
feedback  that  is  neither  reliable  nor  useful  to  the  directorate/department  re¬ 
ceiving  it.  Aside  from  the  methods  themselves,  inadequate  analysis  by  the  DOES 
and  lack  of  direct  feedback  from  users  are  the  factors  that  are  most  detrimental 
to  the  reliability  and  usefulness  of  feedback. 

The  types  of  feedback  discussed  in  this  section  are  listed  in  Table  1, 
along  with  the  types  of  feedback  available  to  the  DTD,  which  are  discussed  in 
the  next  section.  Also  included  in  this  table  are  the  sources  of  feedback  and 
the  initial  recipient  of  each  type  (DOES,  DTD,  or  both).  The  table  is  intended 
to  summarize  both  this  and  the  next  section  and  to  bridge  the  gap  between  them. 
While  reading  the  next  section,  it  may  be  useful  to  refer  back  to  Table  1  to 
obtain  a  summary  of  the  types  of  feedback  already  discussed. 
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SECTION  IV 


DTD  FEEDBACK  SOURCES  AND  NEEDS 

In  this  section  the  feedback  needs  of  training  developers  are  addressed 
through  review  of  the  results  of  interviews  with  the  DTD  division  and  hranch 
chiefs.  Of  the  six  division  chiefs  interviewed,  four  indicated  that  person¬ 
nel  in  their  division  have  a  continuing  need  for  feedback  on  the  performance 
of  soldiers  and  use  of  training  materials.  Two  divisions  are  primarily  in¬ 
volved  in  the  production  and  distribution  of  training  materials  rather  than 
in  their  development.  Personnel  in  these  divisions  have  relatively  little 
direct  need  for  feedback,  and  thus  they  were  not  interviewed  extensively. 

The  information  presented  here  is  derived  primarily  from  interviews  with  20 
personnel  involved  in  task  analysis,  the  design  and  development  of  individual 
and  collective  training  programs  and  materials,  and  the  development  and  manage¬ 
ment  of  training  devices.  As  argued  earlier,  these  personnel  are  representa¬ 
tive  of  those  in  any  Center/School  DTD. 

In  order  to  avoid  attribution  of  interview  responses  to  specific  individ¬ 
uals,  the  information  gathered  is  not  organized  by  division  or  branch  in  this 
section.  Rather,  the  results  are  organized  first  by  types  of  feedback  currently 
available  and  secondly  by  major  feedback  issues  or  needs  that  arose  during  the 
interviews.  In  the  descriptions  of  feedback  currently  available,  emphasis  is 
placed  on  types  of  feedback  from  sources  external  to  the  Center/School  (i.e., 
field  units),  but  internal  feedback  is  also  briefly  addressed.  No  attempt  is 
made  to  describe  these  types  of  feedback  in  great  depth  or  to  detail  the  ways 
in  which  they  are  transmitted  through  the  DTD.  Instead,  brief  descriptions  of 
them  are  given  to  serve  as  a  basis  for  the  follow-on  discussion  of  feedback 
needs.  These  needs  are  described  in  terms  of  major  problems  or  issues  which 
surfaced  from  the  interviews,  and  types  of  feedback  are  further  discussed  and 
evaluated  as  they  relate  to  these  issues.  Recommendations  for  ways  to  satisfy 
the  feedback  needs  identified  are  developed  in  a  later  section  of  this  report. 

Feedback  Currently  Available 

Trip  reports.  A  common  way  in  which  DTD  personnel  obtain  external  feed¬ 
back  is  through  reviewing  written  reports  of  information  gathered  during^visits 
to  field  units.  These  trips  are  usually  taken  and  reports  written  by  DOiiS  per¬ 
sonnel,  but  reports  are  occasionally  received  by  the  DTD  for  trips  taken  by  the 
Commanding  General  or  other  Center/School  personnel.  All  DTD  personnel  inter¬ 
viewed  who  have  a  need  for  feedback  from  the  field  indicated  that  they  receive 
copies  of  DOES  trip  reports  or  extracts  from  them,  as  described  in  the  previous 
section.  While  nest  interviewees  stated  that  useful  indications  of  problems  in 
the  field  are  sometimes  derived  from  trip  reports,  several  criticisms  of  the 
utility  of  such  reports  as  feedback  were  consistently  brought  out. 

The  most  common  critique  of  trip  renorts  mentioned  by  nearly  all  inter¬ 
viewees  is  that  these  reports  do  not  usually  provide  enough  specific  detail 
about  problems  existing  in  the  field  and  recommended  solutions  to  them-  Major 
problems  with  the  quality  of  soldiers  or  training  materials  are  briefly  de- 
scribed  in  trip  reports,  based  upon  observation  of  training,  administration  o 
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surveys  or  interviews,  and  informal  interactions  with  field  unit  personnel. 

But  problems  are  often  not  described  in  sufficient  detail  to  allow  training 
developers  to  determine  what  to  do  about  them.  An  example  of  this  situation 
provided  by  two  interviewees  is  the  statement  from  a  trip  report  that  "lieuten¬ 
ants  can't  read  maps."  Training  developers  need  further  information  in  order 
to  address  this  problem,  such  as  the  time  frame  during  which  these  officers 
completed  institutional  training  and  the  subtasks  which  cause  the  greatest 
difficulty.  With  such  information  available,  training  could  be  redesigned 
to  address  the  problem. 

Several  other  problems  with  the  obtaining  of  feedback  from  trip  reports 
were  mentioned  by  more  than  one  interviewee.  Five  interviewees  expressed 
concerns  about  the  validity  of  information  provided  in  trip  reports.  Trip 
reports  often  summarize  general  observations  or  opinions  which  may  not  be 
completely  accurate.  Trip  reports  also  often  do  not  delineate  the  extent  of 
a  problem  identified.  That  is,  it  is  sometimes  not  clear  whether  a  problem 
is  unique  to  a  particular  unit  or  widespread  throughout  field  units.  It  is 
also  sometimes  unclear  whether  a  perceived  problem  is  based  on  the  opinion 
of  a  few  individuals  or  represents  the  consensus  of  numerous  field  personnel. 

Two  interviewees  brought  up  another  concern  with  the  validity  of  DOES  trip 
reports.  This  is  that  information  gathered  in  the  field  by  personnel  per¬ 
ceived  to  be  evaluators  (i.e.,  DOES  personnel)  may  not  accurately  reflect  the 
state  of  field  units.  At  least  some  units  tend  to  put  their  best  foot  forward 
for  evaluators  and  to  let  these  personnel  see  what  they  want  them  to  see. 

Another  problem  with  extracting  feedback  from  trip  reports  is  related  to 
their  distribution.  While  all  DTD  interviewees  indicated  that  they  had  seen 
DOES  trip  reports,  the  reported  frequency  of  such  reports  varied.  Three  per¬ 
sonnel  reported  that  they  had  not  seen  a  DOES  trip  report  in  over  a  year,  and 
several  others  indicated  that  they  saw  them  irregularly  and  had  not  seen  one 
for  several  months.  Several  interviewees  also  expressed  a  suspicion  that 
trip  reports  prepared  by  personnel  in  agencies  other  than  the  DOES  are  not 
distributed  to  all  personnel  having  an  interest  in  them.  Related  to  concerns 
about  the  distribution  of  trip  reports  are  concerns  about  their  timing. 

Reliance  upon  trip  reports  for  feedback  sometimes  does  not  provide  information 
within  the  time  frame  that  it  is  needed.  For  example,  roost  interviewees  indi¬ 
cated  that  they  have  an  opportunity  to  provide  questions  to  DOES  personnel  to 
be  addressed  during  upcoming  field  visits.  But  one  interviewee  indicated  that 
responses  to  such  questions  are  not  included  in  trip  reports  until  four  to 
eight  months  later,  and  two  indicated  that  responses  have  never  been  received. 
Another  problem  here  is  that  DTD  personnel  may  not  know  what  questions  to  ask 
if  they  do  not  know  what  the  major  concerns  in  the  field  currently  are.  That 
is,  trip  reports  do  not  provide  continuous  feedback  in  which  major  problems  are 
identified  during  initial  trips  and  followed  up  with  more  intensive  questioning 
during  later  trips.  Trip  reports  generally  do  not  provide  data  which  can  be 
quantified  and  maintained  over  time  to  establish  trends  and  to  allow  integration 
of  data. 

The  discussion  above  shows  that,  while  review  of  trip  reports  is  a  common 
and  sometimes  useful  method  of  obtaining  external  feedback,  training  developers 
have  several  concerns  about  the  quality  and  usefulness  of  the  data  acquired. 


The  review  of  trip  reports  is  in  general  a  rather  unsystematic  way  of  gathering 
feedback.  Problems  with  trip  reports  and  ways  of  remedying  them  are  further 
discussed  in  the  context  of  general  issues  in  the  next  section  of  this  report. 

Field  visits.  An  alternative  to  training  developers'  relying  on  reports 
of  trips  by  other  personnel  to  obtain  feedback  would  be  for  them  to  visit  field 
units  themselves.  Interview  results  indicate  that  such  visits  occur  only 
occasionally.  Less  than  half  the  DTD  personnel  interviewed  indicated  that  they 
had  visited  field  units,  and  those  who  had  reported  that  such  visits  were  rare. 
The  number  of  visits  per  branch  varied  from  one  or  two  a  year  to  one  every 
three  or  four  years.  The  DTD  does  not  have  the  resources  for  extensive  field 
visits,  since  that  mission  has  been  primarily  given  to  the  DOES.  The  centrali¬ 
zation  of  field  visits  within  the  DOES  is  an  efficient  use  of  travel  resources 
and  minimizes  the  disruption  of  field  units  by  visits  from  Center/School  per¬ 
sonnel.  However,  restrictions  on  visits  by  training  developers  to  the  field 
does  create  some  problems,  many  of  which  were  described  in  the  earlier  discus¬ 
sion  of  trip  reports. 

Reliance  on  feedback  gathered  during  field  visits  by  other  personnel  re¬ 
sults  in  training  developers  working  with  secondhand  information.  As  described 
earlier,  several  interviewees  indicated  that  this  lessens  the  utility  of  feed¬ 
back,  since  the  information  gathered  is  of  questionable  validity  or  not  in 
sufficient  detail.  A  few  interviewees  indicated  that  they  will  not  accept  any 
feedback  as  accurate  unless  they  gather  it  themselves  directly  from  the  source 
in  the  field.  Whether  this  perception  is  correct  or  not,  it  leads  to  an  atti¬ 
tude  which  does  not  foster  cooperation  between  DTD's  and  DOES's.  Nearly  half 
of  the  DTD  personnel  interviewed  indicated  that  there  is  a  need  for  increased 
personal  contact  between  training  developers  and  users  in  the  field.  Sugges¬ 
tions  for  increased  coordination  of  field  visits  between  training  developers 
and  evaluators  are  offered  in  a  later  section  of  this  report. 

The  few  trips  that  interviewees  had  made  to  the  field  were  perceived  as 
very  useful.  Most  of  these  trips  had  been  taken  by  front-end  analysts  during 
the  early  stages  of  training  development.  Extensive  surveys  and  interviews  of 
field  personnel  were  conducted  during  many  of  these  visits,  although  this  fact 
has  often  been  downplayed  in  order  to  avoid  time-consuming  coordination  of 
survey  forms.  Task  analysts  have  obtained  updated  validation  of  task  documen¬ 
tation  (tasks,  conditions,  and  standards)  during  these  visits,  and  in  most 
cases  they  feel  that  such  information  is  more  valid  than  that  gathered  through 
other  sources  discussed  below. 

If  more  personal  contact  is  needed  between  training  developers  and  users 
in  the  field,  an  obvious  solution  would  be  for  DTD  personnel  to  more  frequently 
visit  the  field.  Seven  of  the  20  interviewees  directly  stated  that  they  need 
to  visit  the  field  more  frequently,  and  several  others  indirectly  indicated 
such  a  need.  Many  potential  advantages  of  such  visits  were  mentioned.  Imple¬ 
mentation  assistance  provided  by  training  developers  might  increase  the  use  of 
training  materials.  Observation  of  task  performance  in  the  field  would  lead 
to  more  objective  validation  of  task  documentation  than  that  obtained  through 
surveys  and  interviews.  Training  developers  would  be  perceived  as  helpers  and 
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would  obtain  more  useful  information  from  field  personnel  than  training  evalu¬ 
ators  do.  But  despite  all  these  potential  benefits,  not  all  interviewees 
favored  increasing  the  number  of  field  visits.  Many  branches  are  understaffed 
and  do  not  have  the  personnel  resources  to  conduct  such  visits.  The  reliability 
of  observations  made  during  field  visits  was  also  questioned.  As  mentioned  ear¬ 
lier,  many  personnel  feel  that  units  let  you  see  only  what  they  want  you  to  see 
during  visits  to  the  field.  Field  visits  must  be  properly  resourced  and  de¬ 
signed  before  their  number  is  increased. 

In  summary,  interview  results  indicate  that  training  developers  rarely  make 
field  visits.  But  when  such  visits  are  made,  the  feedback  derived  from  them  is 
generally  perceived  as  being  very  useful.  There  was  a  widespread  but  not  uni¬ 
versal  feeling  among  interviewees  that  their  opportunities  to  visit  the  field 
should  be  increased.  Methods  are  needed  to  increase  the  interaction  between 
training  developers  and  users  in  the  field  within  existing  resources.  Possible 
approaches  to  this  are  discussed  in  a  later  section  of  this  report. 

CO  DAP  results.  Another  major  type  of  external  feedback  which  is  currently 
used  by  training  developers  is  the  results  of  Comprehensive  Occupational  Data 
Analysis  Program  (CODAP)  surveys.  As  described  earlier,  these  surveys  are  con¬ 
ducted  periodically  by  the  Soldier  Support  Center,  National  Capitol  Region  in 
Alexandria,  VA  and  descriptive  analyses  of  the  results  are  provided  to  Center/ 
School  personnel.  The  surveys  generally  list  a  long  sequence  of  tasks  and  ask 
respondents  to  rate  each  task  on  a  variety  of  factors,  such  as  how  critical  it 
is,  how  frequently  it  is  performed,  and  how  well  it  is  performed.  The  results 
of  such  surveys  are  of  primary  interest  to  task  analysts,  and  these  personnel 
are  directly  involved  in  the  coordination  of  these  surveys  and  use  of  results. 
Task  analysts  determine  the  tasks  to  be  addressed  in  CODAP  surveys,  review  and 
maintain  files  of  computer  print-outs  of  results,  and  request  further  analyses 
of  the  data  from  Soldier  Support  Center  as  needed.  These  analyses  are  primarily 
descriptive  breakdowns  of  the  data  according  to  specified  criteria.  For  example, 
a  listing  can  be  provided  for  all  tasks  meeting  specified  levels  of  performance 
frequency  and  criticality  for  a  given  MOS.  The  results  are  used  by  task  analysts 
and  by  task  selection  boards  to  select  tasks  for  training  in  the  institution  and 
in  field  units.  CODAP  results  thus  provide  feedback  that  can  be  used  to  deter¬ 
mine  what  tasks  should  be  trained  where. 

The  training  developers  interviewed  mentioned  several  concerns  with  the  use 
of  CODAP  results.  A  major  one  is  that  the  results  are  not  timely.  A  CODAP  sur¬ 
vey  addressing  officers’  tasks  within  a  particular  specialty  has  been  conducted 
only  once  approximately  two  years  ago,  and  surveys  of  enlisted  tasks  are  accom¬ 
plished  only  once  every  three  years.  Due  to  the  rapidly  changing  technology  of 
today's  Army,  more  continuous  feedback  on  task  performance  is  needed.  The  per¬ 
sonnel  resources  of  Soldier  Support  Center  would  have  to  be  increased  in  order  to 
provide  such  feedback.  Another  concern  with  CODAP  results  has  to  do  with  their 
perceived  validity.  Some  interviewees  perceived  that  subjective  estimates 
gathered  in  lengthy  surveys  may  not  be  valid  indicators  of  the  importance  of 
tasks  or  the  proficiency  with  which  they  are  performed.  Actual  observation  of 
task  performance  in  the  field  may  be  the  only  way  to  accurately  measure  the  fac¬ 
tors  addressed  in  CODAP  surveys.  For  further  support  of  this  position,  see 


Burnside  (1982).  A  final  major  concern  with  CODAP  data  is  that  they  are  not 
always  available.  No  CQDAP  results  are  available  for  Reserve  units,  and  they 
are  badly  needed.  Results  are  also  not  available  for  some  low-density  MOS's, 
and  for  tasks  performed  on  equipment  which  has  recently  been  introduced  to  the 
field.  Since  18  months  are  required  to  receive  data  after  a  task  list  has  been 
submitted  for  a  CODAP  survey,  needed  data  are  sometimes  not  available.  Or  they 
are  not  available  when  needed,  as  discussed  under  the  timeliness  issue  above. 

The  stated  concerns  about  CODAP  data  can  be  related  to  the  lack  of  personnel 
resources  to  perform  surveys  more  frequently  and  to  check  their  validity,  and 
to  the  lack  of  sufficient  computer  priority  to  perform  analyses  quickly.  CODAP 
surveys  provide  an  economical  but  not  completely  satisfactory  way  of  obtaining 
task-specific  feedback. 

Because  of  their  concerns  with  the  timeliness  and  validity  of  CODAP  re¬ 
sults,  training  developers  sometimes  attempt  to  supplement  these  data.  One  way 
in  which  this  is  done  is  through  field  visits,  as  discussed  earlier.  Visits  of 
DTD  personnel  to  field  units  are  often  called  job  site  visits  or  interviews, 
but  they  actually  often  involve  administration  of  CODAP-type  surveys.  Such  sur¬ 
veys  address  a  smaller  list  of  tasks  than  is  usually  addressed  in  a  CODAP  survey 
and  they  are  frequently  administered  during  an  interview  so  that  the  training 
developer  is  available  to  answer  questions.  Use  of  these  procedures  leads 
training  developers  to  perceive  that  the  results  of  these  interviews  are  more 
valid  than  results  of  CODAP  surveys.  Interviewees  indicated  that  if  a  discrep¬ 
ancy  existed  between  CODAP  results  and  results  of  a  branch-conducted  job  site 
visit,  the  latter  would  be  accepted  as  more  accurate.  Until  the  CODAP  survey 
system  can  be  made  more  flexible  and  timely,  training  developers  will  rely  on 
data  gathered  during  field  visits  whenever  possible.  At  present  there  is  no  way 
of  integrating  these  two  sources  of  data  and  checking  their  relative  validity. 

SQT  item  analysis.  Field  performance  data  are  rarely  available  as  feedback 
to  training  developers.  Most  of  the  currently  available  feedback  identified  by 
training  developers  interviewed  consists  of  general  observations,  informal  com¬ 
ments,  and  subjective  survey  responses.  The  only  relatively  objective  perform¬ 
ance  data  which  have  been  available  as  external  feedback  to  the  DTD  are  Skill 
Qualification  Test  (SQT)  results.  These  hands-on  performance  test  results  are 
provided  to  SQT  developers  in  the  DTD  (and  to  training  evaluators  in  the  DOES, 
as  discussed  previously)  by  TRADOC's  Army  Training  Support  Center  in  the  form 
of  quarterly  and  annual  computer  print-outs,  or  item  analyses.  These  analyses 
provide  feedback  on  the  percentages  of  soldiers  passing  and  failing  each  task 
tested  on  the  hands-on  and  written  components  of  the  SQT.  This  information  is 
used  to  revise  the  test  for  future  administrations.  Tasks  which  are  consistently 
passed  by  100%  of  soldiers  tested  may  be  eliminated  on  future  tests.  Tasks  which 
are  failed  by  a  large  number  of  soldiers  may  also  be  eliminated,  or  the  question 
may  be  revised  if  its  form  appears  to  have  created  the  difficulty  encountered. 

The  item  analyses  provide  important  feedback  to  SQT  developers  as  to  the  adequacy 
of  their  test  construction.  They  also  provide  performance  data  which  could  be  of 
use  to  other  training  developers.  Two  interviewees  not  involved  in  SQT  develop¬ 
ment  indicated  that  they  need  increased  access  to  SQT  results  in  order  to  identi¬ 
fy  potential  training  problems  in  the  field. 
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However,  this  need  for  increased  access  to  item  analyses  throughout  DTD  is 
apparently  a  moot  point.  Current  plans  call  for  formal  administration  of  the 
hands-on  component  of  the  SQT  and  reporting  of  results  to  be  eliminated  by 
FY  83.  Item  analyses  of  hands-on  performance  test  results  will  thus  no  longer 
be  available.  Results  of  written  knowledge  tests  will  continue  to  be  available, 
but  these  results  likely  will  not  as  directly  indicate  where  training  problems 
occur  in  the  field.  The  amount  of  field  performance  data  available  as  feed¬ 
back  to  training  developers  will  soon  decline  from  little  to  none. 

Informal  feedback.  Several  of  the  types  of  feedback  currently  available 
to  the  DTD  can  be  grouped  together  because  they  are  not  regularly  collected 
through  formal  means.  Included  here  are  responses  to  questionnaires  included 
in  the  back  of  training  materials,  requests  for  changes  in  training  materials 
on  DA  Forms  2028,  unsolicited  letters  and  phone  calls  from  field  personnel, 
meetings  at  conferences,  and  information  gathered  due  to  turnover  of  personnel 
in  the  DTD.  A  common  characteristic  shared  by  most  of  these  types  of  feedback 
is  that  they  only  infrequently  provide  useful  information  to  training  develop¬ 
ers.  Almost  all  training  manuals  include  short  questionnaires  for  users  to 
complete  and  return  to  training  developers.  Interviewees  indicated  that  such 
questionnaires  are  seldom  returned;  most  estimates  of  return  rate  were  in  the 
range  of  one  or  two  per  year.  Requests  for  changes  in  training  publications 
are  received  slightly  more  frequently;  interviewees  estimated  that  they  see 
from  two  to  six  per  year.  But  these  suggestions  do  not  usually  result  in 
changes,  since  the  suggestions  are  often  general  in  nature,  do  not  lead  to 
feasible  changes,  or  point  out  problems  which  training  developers  are  already 
aware  of.  If  possible,  DTD  personnel  respond  to  personnel  suggesting  changes 
and  state  what  change  was  made  or  explain  why  no  change  was  appropriate.  The 
low  rate  of  suggestions  from  the  field  should  thus  not  be  due  to  the  attitude 
that  they  will  not  be  attended  to.  Other  causes  may  be  a  low  usage  rate  of 
training  materials  in  the  field,  or  a  hesitancy  to  take  the  time  to  put  com¬ 
ments  into  writing.  Interviewees  also  indicated  that  they  receive  unsolicited 
letters  and  phone  calls  from  field  personnel  at  the  rate  of  two  or  less  per 
year.  Field  personnel  for  some  reason  do  not  frequently  use  informal  channels 
to  communicate  feedback  to  training  developers.  Meetings  at  conferences  also 
do  not  contribute  much  feedback.  Several  interviewees  indicated  that  confer¬ 
ences  provide  only  general  feedback  which  is  usually  not  useful.  During  20 
interviews  only  one  instance  of  useful  feedback  being  obtained  during  a  con¬ 
ference  was  reported.  This  occurred  when  a  battalion  commander  attending  a 
conference  actively  sought  out  personnel  in  a  branch  of  the  DTD  and  provided 
specific  feedback  on  use  of  specific  training  materials  in  his  unit.  Methods 
are  needed  to  increase  the  frequency  with  which  such  feedback  is  provided. 
Reliance  on  approaches  as  unstructured  as  those  described  in  this  paragraph 
is  unlikely  to  produce  this  result. 

A  type  of  informal  feedback  which  more  frequently  provides  useful  informa¬ 
tion  Is  based  upon  the  turnover  of  military  personnel  within  the  DTD.  Officers 
and  NCO's  transferring  into  the  DTD  often  come  from  an  assignment  in  the  field. 
They  thus  bring  with  them  useful  information  on  the  quality  of  soldiers  and 
use  of  training  materials  in  the  field.  This  is  a  benefit  of  the  high  turbu¬ 
lence  rate  in  the  military  assignment  system  and  is  a  way  to  increase  congruence 
between  training  developers  and  field  personnel.  However,  the  information 
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gathered  by  this  method  is  limited  by  human  memory  and  by  the  experiences  which 
training  developers  had  while  assigned  to  field  units.  This  type  of  feedback 
is  an  important  one,  but  it  should  be  supplemented  by  information  gathered 
regularly  in  a  wide  variety  of  units. 


Internal  feedback.  The  types  of  feedback  described  thusfar  in  this 
section  are  external  ones  relating  to  soldier  performance  and  use  of  training 
materials  in  the  field.  Training  developers  also  receive  internal  feedback 
relating  to  institutional  training  from  other  directorates  and  departments  in 
the  Center /School .  A  large  amount  of  such  feedback  is  based  upon  informal 
exchange  of  information  between  personnel  in  DTD  and  other  agencies,  but 
several  more  formal  types  of  internal  feedback  are  available.  Instructional 
departments  submit  requests  for  changes  in  Programs  of  Instruction  (POI's)  to 
training  developers.  Such  requests  include  a  rationale  for  why  changes  are 
needed,  and  thus  provide  feedback  on  where  problems  lie  in  training.  Student 
critique  sheets  or  other  surveys  are  usually  completed  at  the  conclusion  of 
courses,  and  results  are  available  as  feedback  to  the  DTD  through  the  DOES,  as 
described  earlier.  However,  several  of  the  DTD  personnel  interviewed  stated 
that  they  do  not  consider  this  an  important  form  of  feedback,  due  to  concerns 
about  the  validity  of  students'  comments.  The  general  feeling  is  that  students 
comments  are  influenced  more  by  the  personality  or  presentation  style  of  the 
instructor  than  by  the  content  of  the  course.  Training  developers  are  more 
likely  to  take  stock  in  comments  received  from  instructors  than  in  those  received 
from  students. 


Several  procedures  followed  by  training  developers  during  design  of  training 
programs  and  materials  could  be  described  as  gathering  of  internal  feedback. 
Experts  on  various  tasks  at  the  Center/School  are  often  informally  sought  out  and 
consulted  during  training  development.  For  example,  if  training  on  a  chemical 
task  is  included  in  a  training  program,  local  experts  on  such  a  task  will  be 
identified  and  asked  to  review  the  task  documentation.  After  a  training  program 
is  initially  developed,  it  may  be  pilot  tested  through  student  trials  or  pilot 
teachings.  Student  trials  involve  administration  of  a  part  of  the  training 
program  to  a  small  group  of  students  available  on  a  temporary  basis.  Pilot 
teachings  are  the  first  administration  of  a  new  or  revised  training  program  to  a 
regular  group  of  students.  DTD  personnel  are  directly  involved  in  both  of  these 
procedures  and  useful  feedback  on  the  adequacy  of  course  design  is  gathered 
through  observation  and  interviewing  of  students.  The  information  gathered  falls 
under  the  general  rubric  of  feedback,  although  it  relates  to  the  evaluation  of 
initial  or  draft  products  rather  than  finished  products.  DTD  personnel  seldom 
have  the  opportunity  to  observe  the  administration  of  a  training  program  after 
the  pilot  teaching.  Such  observations  are  performed  by  DOES  personnel,  and  re¬ 
ports  of  results  have  in  the  past  been  provided  to  DTD  on  a  quarterly  basis  (as 
mentioned  in  a  previous  section,  such  reports  are  not  currently  being  Provided 
to  DTD),  tost  interviewees  indicated  that  they  receive  such  reports  and  find 
them  useful;  the  results  of  performance  tests  conducted  during  training  were 
frequently  pointed  out  as  being  especially  useful  information. 


Summary.  Several  types  of  feedback  are  currently  available  to  DTD  person¬ 
nel,  and" "roost  of  these  have  been  described  above  and  were  previously  summarized 
in  Table  1.  Other  types  of  information  are  available  which  could  be  constru 
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as  feedback,  but  do  not  merit  separate  discussion.  For  example,  training 
developers  receive  comments  during  staffing  of  training  programs  and  materials 
with  Center/School  and  TRADOC  agencies.  However,  such  comments  do  not  fit  the 
strict  definition  of  feedback  since  they  relate  to  draft  products  rather  than 
to  finished  products  that  have  been  used  in  rhe  field.  Also,  personnel  inter¬ 
viewed  in  DTD  indicated  that  generally  few  comments  are  received  during  staffing 
training  materials.  Specific  survey  and  data  collection  forms  used  by  the 
DOES  to  gather  feedback  were  described  in  an  earlier  section  of  this  report  and 
have  not  been  further  discussed  here.  For  example,  DOES  personnel  administer  a 
survey  to  field  personnel  on  the  use  of  ARTEP  materials  and  provide  the  results 
directly  to  ARTEP  developers  in  DTD.  The  intent  of  this  section  was  to  provide 
a  general  description  of  feedback  available  to  serve  as  a  basis  for  detailed 
discussion  of  feedback  needs  below. 

While  many  of  the  types  of  feedback  available  to  training  developers  are 
very  useful,  several  problems  are  apparent  in  the  flow  of  feedback  information. 
In  general,  the  flow  of  feedback  appears  to  be  informal,  unsystematic,  and 
based  largely  upon  subjective  data.  Key  issues  and  needs  are  identified  and 
discussed  across  types  of  feedback  below,  in  order  to  develop  recommendations 
for  improving  feedback  flow. 

Major  Feedback  Needs 

Lack  of  feedback.  The  most  general  conclusion  about  the  feedback  avail¬ 
able  to  training  developers  is  that  there  isn't  enough  of  it.  All  20  inter¬ 
viewees  who  indicated  a  need  for  feedback  from  the  field  also  indicated  that 
they  presently  receive  an  insufficient  amount  of  it.  They  feel  that  they  as 
training  developers  should  be  more  directly  involved  in  gathering  feedback  and 
should  have  more  direct  contact  with  personnel  in  the  field.  Changes  occur  in 
the  field  at  a  rapid  pace  and  training  developers  have  difficulty  keeping  in¬ 
formed  of  them.  Problems  are  often  surfaced  through  informal  channels  or 
rumors  and  training  developers  may  not  have  sufficient  time  to  react  to  them. 

Interviewees  did  not  present  a  consensus  on  how  the  flow  of  feedback 
should  be  increased.  Some  stated  that  they  should  visit  field  units  and  provide 
assistance  in  implementation  of  new  training  programs  and  materials.  Others 
indicated  that  increased  travel  to  the  field  is  impractical  and  that  field  per¬ 
sonnel  should  somehow  be  further  encouraged  to  contact  training  developers.  It 
might  be  more  cost-effective  to  bring  experienced  field  personnel  to  the  Center/ 
School  than  to  send  DTD  personnel  to  the  field.  Field  personnel  temporarily 
assigned  to  the  Center/School  to  attend  a  training  course  could  also  be  further 
used  as  a  feedback  source  than  they  presently  are.  These  individuals  could  be 
surveyed  or  interviewed  about  specific  areas  of  concern  upon  their  arrival  at 
the  Center /School.  As  described  earlier,  these  personnel  currently  complete  a 
general  survey  during  their  first  two  weeks  at  the  Center/School,  but  they  are 
not  interviewed  in  depth.  Several  interviewees  expressed  the  opinion  that  feed¬ 
back  on  training  materials  is  rarely  received  from  the  field  because  these 
materials  are  in  fact  seldom  used.  The  first  step  in  gathering  feedback  should 
thus  be  to  survey  the  field  on  the  availability  and  use  of  training  materials. 
The  same  argument  can  be  made  for  training  devices.  These  and  other  suggestions 
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for  increasing  the  amount  of  feedback,  available  to  the  DTD  will  be  used  to 
develop  more  specific  recommendations  in  a  later  section  of  this  report. 


Specificity  of  feedback.  A  common  observation  offered  by  almost  all 
interviewees  is  that  much  of  the  feedback  available  is  not  sufficiently 
detailed  or  specific.  A  general  problem  may  be  identified,  but  details  are 
usually  not  provided  which  isolate  the  exact  cause  of  the  problem  and  ways  to 
correct  it.  This  situation  exists  with  trip  reports  received  from  DOES  and 
other  sources  and  with  most  of  the  informal  feedback  received.  If  field 
personnel  indicate  that  graduates  of  institutional  training  are  not  well 
trained  or  that  training  materials  are  not  useful,  training  developers  need 
further  information.  They  need  to  know  which  specific  tasks  or  subtasks  are 
causing  performance  difficulties  and  which  parts  of  training  materials  are 
not  being  used  and  why.  As  in  the  example  used  earlier,  training  developers 
need  to  know  not  only  that  lieutenants  can't  read  maps,  but  which  lieutenants 
have  difficulty  with  which  aspects  of  map  reading.  Training  programs  can 
then  be  redesigned  to  attempt  to  remedy  the  problem. 

DTD  personnel  rely  heavily  upon  DOES  and  other  agencies  to  gather  feed¬ 
back  for  them,  or  upon  field  personnel  to  volunteer  feedback.  This  leads  to 
the  provision  of  feedback  without  satisfactory  detail.  It  is  often  tedious 
and  time-consuming  for  training  developers  to  go  back  to  the  source  of  the 
feedback  and  obtain  further  information.  One  solution  to  this  problem  would 
be  for  these  personnel  to  be  directly  involved  in  the  original  gathering  of 
feedback.  Further  details  could  then  be  gathered  directly  from  the  feedback 
source  in  one  operation.  Another  solution  would  be  to  increase  the  coordina¬ 
tion  between  those  who  need  the  feedback  in  the  DTD  and  those  who  gather  it 
in  the  DOES.  DTD  personnel  have  the  opportunity  to  submit  specific  questions 
to  DOES  personnel  before  field  units  are  visited,  but  this  coordination  is 
apparently  not  sufficient.  Training  developers  may  not  know  what  questions  to 
ask,  since  they  have  not  received  previous  feedback  as  to  what  the  primary 
concerns  of  the  field  are.  Interviewees  indicated  that  in  the  past  they 
have  not  received  answers  to  questions  submitted  within  the  needed  time  frame 
or  at  all.  This  perceived  lack  of  response  does  not  encourage  submission  of 
future  questions.  Even  if  DOES  personnel  do  ask  questions  developed  by  DTD 
personnel  in  the  field,  they  may  not  have  sufficient  experience  with  the  par¬ 
ticular  document  or  device  being  addressed  to  get  the  needed  details  by  asking 
appropriate  follow-on  questions.  The  only  solution  to  this  dilemma  would  be 
for  training  developers  to  accompany  training  evaluators  on  visits  to  field 
units. 

Validity  of  feedback.  Several  of  the  training  developers  interviewed 
expressed  concerns  about  whether  the  information  they  receive  as  feedback  is 
valid  and  reliable.  It  is  frequently  difficult  to  determine  how  representative 
feedback  is  of  the  situation  in  the  field  as  a  whole.  For  example,  a  problem 
with  a  particular  document  may  be  identified  through  comments  in  a  trip  report 
or  through  informal  means,  such  as  a  phone  call  from  an  officer  in  the  field  or 
a  comment  from  an  NCO  newly  assigned  to  the  DTD.  Is  this  prohlem  widespread 
throughout  field  units  or  does  it  represent  the  experiences  of  one  individual 
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in  just  one  unit?  There  is  at  present  no  structured  way  to  answer  this  ques¬ 
tion.  When  a  problem  is  identified,  methods  are  needed  to  pursue  it  in 
future  collection  of  feedback  in  order  to  determine  its  extent.  (Hear  criteria 
should  be  established  as  to  which  and  how  many  personnel  should  be  surveyed  or 
interviewed  during  field  visits,  and  as  to  how  many  times  a  problem  should 
surface  before  it  is  considered  severe  enough  to  require  action.  Formal  and 
informal  means  of  feedback  collection  should  be  coordinated  so  that  informal 
feedback  provides  input  into  areas  to  be  further  pursued  by  more  formal 
methods. 

Much  of  the  feedback  presently  received  by  training  developers  is  based 
upon  subjective  opinions  of  personnel  in  the  field.  This  is  true  of  informa¬ 
tion  gathered  through  trip  reports,  CODAP  surveys,  and  informal  methods.  Many 
interviewees  expressed  concern  with  the  validity  of  such  information  and  indi¬ 
cated  a  need  for  more  feedback  based  upon  performance  testing  and  observation 
of  field  performance.  However,  several  of  them  also  expressed  reservations 
about  the  validity  of  field  observations,  based  on  the  feeling  that  field  units 
only  let  you  see  what  they  want  you  to  see.  Few  performance  test  results  are 
presently  available,  and  fewer  will  be  available  in  the  near  future.  SQT  re¬ 
sults  and  results  of  tests  given  during  institutional  training  represent  the 
only  performance  results  available  as  feedback  at  present.  The  hands-on 
performance  test  portion  of  the  SQT  is  scheduled  to  go  by  the  wayside  in  the 
next  few  months,  so  no  performance  results  will  be  available  as  external  feed¬ 
back  at  that  time.  All  these  facts  make  it  difficult  to  alleviate  training 
developers'  concerns  about  the  validity  of  feedback. 

The  only  realistic  way  to  increase  the  validity  of  feedback  is  to  base  it 
further  upon  observations  in  the  field.  Observers  must  be  trained  to  make 
thorough  and  objective  observations  which  are  not  unduly  influenced  by  opinions 
and  interpretations.  Observers  should  also  present  themselves  as  assistants 
and  facilitators  to  the  field,  rather  than  as  evaluators.  Useful  feedback  can 
then  be  gathered  through  observations.  The  only  other  way  to  increase  the 
availability  of  valid  feedback  would  be  to  increase  the  performance  testing  of 
field  personnel.  This  alternative  is  considered  unrealistic,  since  the  Army 
is  currently  moving  away  from  the  testing  of  individual  skills  in  the  field. 
However,  if  the  validity  of  feedback  is  ever  to  be  maximized,  it  will  have  to 
be  done  through  performance  testing. 

Timeliness  of  feedback.  Another  major  problem  that  surfaced  during  the 
interviews  is  that  many  types  of  feedback  are  not  timely.  By  the  time  that 
information  is  received,  the  situation  in  the  field  may  have  changed  and  it 
may  be  too  late  for  training  developers  to  react.  Weeks  or  even  months  may 
go  by  before  information  is  summarized  in  trip  reports  and  distributed  to  those 
who  need  it.  One  interviewee  stated  that  it  has  in  the  past  taken  as  long  as 
eight  months  to  receive  answers  to  questions  asked  in  the  field  through  the 
DOES.  In  this  time  frame  the  question  designer  may  have  been  transferred  or 
may  nc  longer  have  need  for  the  feedback,  since  he  or  she  was  required  to 
complete  actions  without  it.  CODAP  surveys  of  enlisted  tasks  within  a  par¬ 
ticular  MOS  are  conducted  every  third  year,  and  surveys  of  officers'  tasks 
have  thusfar  been  one-time  special  exercises.  Task  analyses  can  thus  be  based 
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on  information  that  is  as  much  as  three  years  old.  Given  the  rapidly  changing 
state  of  technology  in  today's  Army,  training  materials  may  he  out-of-date  when 
they  are  developed.  Item  analyses  of  SCJT  results  are  another  example  of  un¬ 
timely  feedback..  These  analyses  have  in  the  past  not  been  received  in  time  to 
be  used  in  development  of  the  next  iteration  of  tests. 

If,  as  suggested  earlier,  training  developers  were  more  directly  involved 
in  the  collection  of  feedback,  its  timeliness  should  improve.  Direct  contact 
with  the  field  would  allow  training  developers  to  determine  whether  the  informa¬ 
tion  they  are  using  is  current,  More  frequent  administration  of  CODAP  surveys 
would  also  provide  more  up-to-date  information  to  training  developers.  If 
this  is  not  possible,  training  developers  should  be  provided  resources  to  visit 
the  field  themselves  and  supplement  CODAP  information  as  needed.  Feedback  will 
not  be  useful  unless  steps  such  as  these  are  taken  to  improve  its  timeliness. 

Continuity  of  feedback.  Several  interviewees  made  the  comment  that  feed¬ 
back  needs  to  be  more  continuous  than  it  presently  is.  An  example  of  the  non¬ 
continuity  of  feedback  is  the  administration  of  CODAP  surveys  discussed  above. 
These  surveys  are  conducted  every  three  years  and  no  updated  information  is 
gathered  between  administrations.  This  leads  to  the  lack  of  timely  information 
discussed  above,  but  problems  with  the  continuity  of  feedback  are  more  extensive 
than  that.  Different  types  of  feedback  are  often  collected  in  Isolation  and  are 
not  integrated  with  other  types.  For  example,  if  a  problem  in  the  field  is  noted 
in  a  trip  report  or  in  informal  feedback,  it  should  be  further  pursued  in  future 
field  visits  and  interviews.  At  present  there  is  no  system  for  collating  and 
integrating  feedback  and  determining  what  its  major  implications  are.  No  one 
person  or  group  at  the  Center/School  integrates  feedback  previously  gathered  from 
all  sources  and  determines  what  issues  should  be  addressed  in  future  collections 
of  feedback.  If  a  problem  is  identified  during  feedback,  there  is  no  mechanism 
to  follow  it  up  and  insure  that  it  is  solved.  Much  of  this  situation  is  due  to  a 
lack  of  capability  to  analyze  and  maintain  feedback  information  over  time.  This 
problem  is  further  discussed  below. 

Analysis  and  management  of  feedback.  DTD  personnel  presently  have  no  auto¬ 
mated  capability  to  analyze  and  maintain  the  feedback  they  receive.  In  some 
cases  other  agencies  perform  a  descriptive  analysis  of  data  before  providing 
them  to  the  DTD.  For  example,  computer  print-outs  of  descriptive  analyses  of 
CODAP  and  SQT  results  are  available.  But  in  other  cases,  such  as  data  gathered 
during  field  visits  or  through  some  of  the  surveys  administered  by  DOES  person¬ 
nel,  the  only  available  method  of  analysis  is  by  hand.  This  is  a  tedious  pro¬ 
cess  and  the  result  is  that  data  are  sometimes  not  thoroughly  analyzed  and  used. 
For  example,  completed  surveys  are  sometimes  glanced  at  and  thrown  in  a  file 
drawer  without  ever  being  analyzed.  The  lack  of  an  automated  data  storage  and 
retrieval  capability  leads  to  lack  of  capability  to  establish  trends  in  feedback 
over  time.  Different  types  of  feedback  are  thus  not  used  in  an  integrated  and 
continuous  fashion,  as  was  discussed  above. 

If  procedures  are  developed  to  increase  the  flow  of  feedback  to  training  de¬ 
velopers,  procedures  must  simultaneously  be  developed  to  analyze  and  maintain  the 
data  acquired.  An  automated  system  is  needed  which  can  be  jointly  accessed  by 
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collectors  of  feedback  in  the  DOES  and  users  in  the  DTD.  Existing  data  manage¬ 
ment  packages  can  probably  be  modified  for  this  purpose.  But  several  decisions 
about  system  design  need  to  be  made,  such  as  what  data  are  to  he  entered  into 
the  system,  how  data  are  to  be  displayed  in  a  simple,  useful  manner,  and  what 
criteria  should  be  established  to  determine  when  data  indicate  a  significant 
problem  in  the  field.  Many  of  the  problems  with  feedback  discussed  in  this 
paper  cannot  be  solved  without  development  of  an  appropriate  data  analysis 
and  management  system. 

Flow  of  feedback.  A  final  problem  with  feedback,  and  one  which  can  be 
directly  related  to  lack  of  a  data  management  system,  is  the  fact  that  feedback 
is  not  always  available  to  those  who  need  it.  Several  of  the  training  developers 
interviewed  expressed  the  suspicion  that  they  do  not  receive  all  the  available 
feedback  which  might  be  of  interest  to  them.  The  circulation  of  trip  reports 
appears  to  be  rather  spotty.  Some  interviewees  indicated  that  they  see  DOES 
trip  reports  regularly,  while  others  indicated  that  they  had  not  seen  one  in 
over  a  year.  SQT  item  analyses  are  available  to  SQT  developers,  but  other 
training  developers  indicated  that  these  results  are  not  readily  available  to 
them.  Results  of  surveys  on  the  use  of  particular  training  materials  have  in 
the  past  been  available  to  writers  of  the  materials  but  not  to  task  analysts 
and  others  involved  in  the  production  of  the  materials.  So  feedback  does  not 
always  flow  to  all  those  who  feel  a  need  for  it. 

Several  interviewees  attributed  feedback  flow  problems  to  the  current  organ¬ 
izational  structure  of  the  DTD.  At  present,  the  task  analysts,  writers  of  train¬ 
ing  materials,  and  producers  of  final  copies  of  training  materials  are  organized 
into  different  divisions  of  the  DTD.  And  the  evaluators  of  training  materials 
are  located  in  another  directorate,  the  DOES.  Such  an  organizational  structure 
sometimes  leads  to  communications  problems.  For  example,  task  analysts  who 
determine  what  tasks  are  to  be  taught  in  the  institution  do  not  always  have  con¬ 
tinuous  interaction  with  personnel  who  develop  the  training  programs  for  these 
tasks.  One  suggestion  which  was  offered  by  two  interviewees  to  alleviate  this 
problem  is  to  reorganize  the  DTD  by  MOS.  A  given  group  would  then  have  total 
responsibility  for  developing  training  for  that  MOS,  from  task  analysis  through 
training  evaluation.  It  is  not  within  the  purview  of  this  paper  to  suggest  a 
total  reorganization  of  the  DTD.  However,  some  action  is  needed  to  increase  the 
interaction  among  the  divisions  of  the  DTD  and  between  the  DTD  and  the  DOES. 

Summary.  The  major  problems  with  feedback  identified  by  the  DTD  personnel 
interviewed  are  that  there  is  not  enough  of  it,  it  is  often  not  specific  or  timely 
enough,  it  does  not  flow  to  all  those  who  need  it  in  a  continuous  fashion,  its 
validity  is  frequently  undetermined,  and  there  are  no  efficient  ways  to  analyze 
and  manage  it.  The  key  to  meeting  the  feedback  needs  of  DTD  personnel  appears  to 
be  the  regular  collection  of  valid  objective  data  which  can  be  analyzed  and  main¬ 
tained  so  that  they  are  easily  accessible  to  all  potential  users.  Meeting  this 
goal  will  require  the  expenditure  of  more  resources  than  are  now  devoted  to  the 
collection  and  management  of  feedback.  In  the  next  section  the  conclusions 
reached  from  interviews  of  DTD  and  DOES  personnel  are  integiated  and  more  spe¬ 
cific  recommendations  are  developed  for  efficient  use  of  feedback  resources. 
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SECTION  V 


CONCLUSIONS  AND  RECOMMENDATIONS 

While  some  useful  feedback  is  presentlv  available  to  training  developers 
and  evaluators,  the  interview  results  discussed  in  the  previous  two  sections 
suggest  that  there  are  many  ways  in  which  the  flow  of  feedback  information 
needs  to  be  improved.  Conclusions  about  present  feedback  methods  and  proce¬ 
dures  are  discussed  in  this  section,  along  with  recommendations  on  how  to 
improve  them.  The  recommendat ions  are  general  ones  which  should  apply  to  any 
Center/School,  and  not  just  to  the  one  in  which  interviews  were  conducted. 
While  it  is  necessary  to  discuss  the  recommendations  separately  here,  they  are 
interrelated  and  should  be  implemented  in  an  integrated  fashion. 

Feedback  Needs 


The  most  general  conclusion  that  can  be  reached  about  the  present  feed¬ 
back  system  is  that  more  feedback  is  needed.  This  conclusion  is  based  primarily 
upon  the  perspective  of  training  developers  in  the  DTD,  who  almost  unanimously 
stated  that  they  are  not  getting  all  the  feedback  they  need  for  improving 
training  programs  and  materials.  Training  evaluators  in  the  DOES  did  not  per¬ 
ceive  a  widespread  need  for  more  feedback,  and  some  of  them  indicated  that  they 
are  already  receiving  more  feedback  than  they  can  handle.  The  users  of  feedback 
want  more  of  it  and  the  primary  providers  of  feedback  feel  that  they  are  provid¬ 
ing  all  they  can,  given  their  current  resources.  Possible  solutions  to  this 
dilemma  include  involving  training  developers  more  directly  in  the  collection  of 
feedback  and  Increasing  the  coordination  between  training  developers  and  evalu¬ 
ators.  These  and  other  approaches  to  increasine  the  feedback  available  are 
discussed  later  in  this  section. 

The  paucity  of  feedback  available  to  training  developers  in  some  areas 
indicates  that  they  are  correct  in  their  assertion  that  more  feedback  is  needed. 
In  some  cases  feedback  is  needed  where  it  is  not  currently  available,  and  in 
other  cases  feedback  is  already  available,  but  more  specific  information  is 
needed.  The  types  of  feedback  needed  that  are  not  currently  available  include 
objective  performance  results  and  information  on  the  use  of  training  materials. 
Several  of  the  training  developers  interviewed  stated  a  need  for  more  objective 
performance  results,  such  as  results  of  hands-on  tests  of  individual  skills  and 
observations  of  unit  exercises.  Few  such  data  are  currently  available;  the  only 
types  identified  during  the  interviews  were  results  of  the  hands-on  portion  of 
SQT's  and  results  of  institutional  testing  included  in  quarterly  reports  pre¬ 
pared  by  the  DOES.  Hands-on  SQT  results  will  likely  soon  no  longer  be  available, 
and  indications  are  that  institutional  test  results  will  no  longer  be  reported  by 
the  DOES  in  a  regular  formal  fashion.  Training  developers  will  thus  be  receiving 
fewer  objective  performance  results  rather  than  more.  Possible  ways  to  reverse 
this  trend  will  be  addressed  in  a  later  discussion  of  the  validity  of  feedback. 

Another  pressing  need  for  feedback  that  is  not  currently  being  satisfied 
is  the  requirement  for  more  feedback  from  the  field  on  the  availability  and  use 
of  training  materials.  Nearly  all  DTD  personnel  interviewed  indicated  a  need 
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for  increased  feedback  on  the  use  of  training  materials  they  produce.  Inter¬ 
views  with  DOES  personnel  indicated  that,  while  some  of  them  are  aware  of  this 
need,  very  little  has  been  done  to  address  it.  As  a  first  step,  DTD  personnel 
with  DOES  assistance  could  devise  a  generic  set  of  questions  that  could  be 
used  to  address  a  wide  variety  of  training  materials.  From  this  pool  of  ques¬ 
tions,  training  developers  could  select  and  modify  appropriate  questions  to 
determine  the  availability  and  use  of  specific  training  materials  in  the  field. 
As  mentioned  earlier,  the  scarcity  of  feedback  on  training  materials  may  be  due 
to  the  fact  that  they  are  not  received  or  used  in  the  field.  These  general 
issues  should  be  addressed  first,  followed  by  more  specific  questions  on  the 
design  of  training  materials.  In  this  way,  training  developers  would  be  more 
likely  to  obtain  the  information  they  need  to  revise  or  upgrade  the  training 
materials  they  produce. 

The  biggest  problem  with  the  feedback  currently  available  from  training 
evaluators  and  other  sources  is  that  it  tends  to  be  too  general  for  training 
developers'  needs.  Statements  to  the  effect  that  lieutenants  cannot  read  maps 
or  mechanics  cannot  troubleshoot  are  worth  very  little  to  training  developers. 

To  change  the  training  given  at  the  Center/School  in  map  reading  or  trouble¬ 
shooting,  more  information  is  needed.  As  discussed  earlier,  training  develop¬ 
ers  need  to  know  which  soldiers  are  having  difficulty  with  which  aspects  of  the 
task  of  concern.  They  also  need  to  know  the  size  of  the  sample  and  the  methods 
used  in  data  collection,  so  that  the  extent  of  the  problem  can  be  determined. 

There  are  several  ways  in  which  the  specificity  of  feedback  can  be  in¬ 
creased,  all  of  which  involve  increasing  the  interaction  between  DTD  and  DOES 
personnel.  So  that  more  specific  questions  might  be  developed,  DOES  personnel 
should  "extensively"  coordinate  their  data  collection  plans  and  forms  with  DTD 
personnel  before  making  field  trips.  The  inclusion  of  DTD  personnel  in  data 
collection  teams  would  make  available  knowledgeable  personnel  who  could  ask 
specific  follow-on  questions  during  interviews.  A  more  continuous  feedback 
system  than  presently  exists  would  provide  for  maintenance  of  data  across 
field  visits.  General  problem  areas  uncovered  in  one  visit  could  then  be  ad¬ 
dressed  in  more  specific  detail  in  later  visits.  These  possibilities  are  fur¬ 
ther  addressed  in  the  discussion  below  of  coordination  between  training  develop¬ 
ers  amd  evaluators. 


Coordination  Between  Training  Developers  And  Evaluators 


A  properly  functioning  feedback  system  can  be  described  as  a  continuous 
loop.  When  training  developers  and  evaluators  are  not  the  same  people,  as  is 
the  case  in  most  TRADQC  Centers/ Schools,  continuous  coordination  is  required 
along  this  loop.  Developers  of  training  programs  and  materials  should  provide 
input  to  training  evaluators  as  to  what  feedback  is  needed.  The  evaluators 
should  then  collect  the  needed  information  and  provide  it  to  the  developers, 
along  with  at  least  preliminary  analysis  and  interpretation  of  the  data.  De¬ 
velopers  should  then  inform  evaluators  as  to  what  was  done  with  the  feedback, 
what  changes  were  made  in  training  programs  and  materials,  and  what  new  feed¬ 
back  is  needed.  Continuance  of  this  cycle  should  insure  that  training  devel¬ 
opers  receive  the  feedback  that  they  need.  Unfortunately,  the  interview 
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results  indicate  that  the  feedback  process  does  not  operate  through  a  contin¬ 
uous  cycle  in  the  real  world. 

The  interviews  revealed  many  examples  of  the  need  for  increased  coordina¬ 
tion  between  training  developers  and  evaluators.  DTD  personnel  appeared  to 
be  unaware  of  many  of  the  types  of  feedback  that  are  collected  by  the  DOES  and 
several  of  them  expressed  a  hesitancy  to  accept  as  valid  any  data  with  which 
they  were  not  involved  in  collecting.  DOES  personnel  appeared  to  be  unaware 
of  the  types  of  feedback  needed  by  the  DTD,  and  in  many  cases  they  did  not  know 
how,  or  even  if,  DTD  used  the  feedback  that  they  provided.  As  a  more  specific 
example,  DOES  personnel  obtain  subjective  estimates  of  task  performance  pro¬ 
ficiency,  frequency,  and  criticality  during  graduate/supervisor  surveys  admin¬ 
istered  in  the  field  and  feedback  seminars  conducted  at  the  Center/School.  DTD 
personnel  who  have  a  need  for  such  data  did  not  indicate  an  awareness  of  what 
the  DOES  collects,  and  they  sometimes  visit  field  units  to  collect  similar  data 
themselves.  As  a  result,  the  similar  sorts  of  data  collected  through  graduate/ 
supervisor  surveys,  feedback  seminars,  visits  of  DTD  personnel  to  the  field,  and 
CODAP  surveys  are  not  integrated. 

There  are  several  ways  in  which  the  coordination  between  training  developers 
and  evaluators  can  be  increased.  DOES  personnel  should  increase  their  efforts 
to  involve  the  DTD  in  all  phases  of  the  feedback  process,  particularly  in  the 
development  of  questions  and  methods  to  be  used  in  obtaining  data  from  the  field. 
As  was  suggested  earlier,  DTD  personnel  could  develop  pools  of  questions  for 
assessing  the  availability  and  use  of  training  materials.  The  DTD  or  instruc¬ 
tional  departments  could  develop  pools  of  questions  for  other  purposes,  such  as 
assessing  the  performance  of  soldiers  or  the  distribution  and  use  of  training 
devices.  As  soon  as  the  DOES  has  a  tentative  schedule  for  the  units  to  be 
visited  during  a  given  quarter,  the  schedule  should  be  shared  with  the  DTD,  so 
that  DTD  personnel  have  ample  time  to  add  non-standard  items  of  current  interest 
to  the  data  collection  plan.  DOES  personnel  presently  coordinate  their  field 
visits  with  the  DTD  and  other  departments,  but  the  present  informal  system  of 
sending  a  request  to  the  DTD  for  questions  just  prior  to  a  field  visit  appears 
to  be  inadequate.  If  DTD  personnel  have  not  received  previous  feedback  from  the 
DOES,  they  may  not  know  the  appropriate  questions  to  ask.  The  development  of 
pools  of  questions  as  suggested  above  might  help  here,  since  questions  could  be 
selected  from  these  pools  rather  than  being  developed  anew  before  each  set  of 
field  visits.  It  may  be  necessary  for  the  DOES  to  establish  points  of  contact 
at  the  action  officer  level  in  the  DTD  and  departments  through  which  field 
visitations  would  be  announced,  data  collection  needs  would  be  transmitted,  and 
responses  to  these  needs  would  be  returned.  DOES  personnel  could  also  meet  with 
DTD  and  departmental  personnel  prior  to  a  visit  to  map  out  what  they  expect  to 
accomplish  during  the  visit.  This  involvement  of  training  developers  in  data 
collection  could  go  so  far  as  to  include  these  personnel  in  teams  visiting  the 
field;  this  possibility  is  further  discussed  below.  Meetings  between  DOES  and 
DTD  personnel  following  as  well  as  prior  to  a  visit  should  also  be  beneficial. 

To  insure  that  feedback  is  useful  to  those  to  whom  it  is  provided,  both 
training  evaluators  and  developers  should  take  steps  to  close  the  feedback 
loop.  The  preceding  paragraph  discussed  the  necessity  for  training  developers' 


input  into  the  design  of  feedback  questions  and  methods.  In  this  way  the  data 
needed  by  training  developers  are  more  likely  to  be  gathered.  The  DOES  must 
then  provide  feedback  to  the  DTD  on  each  question  asked  and  on  reasons  why  any 
requested  data  were  not  collected.  Where  specific  problems  are  identified  dur¬ 
ing  an  on-site  visit,  training  evaluators  should  provide  recommendations  on  how 
these  problems  might  be  addressed.  Although  such  recommendations  need  not  be 
binding  upon  training  developers,  these  personnel  should  always  inform  the 
training  evaluators  of  actions  taken  to  address  the  problems  identified.  If 
the  problem  is  not  adequately  documented  to  enable  the  training  developer  to 
address  it,  the  training  evaluator  should  be  so  advised.  In  these  ways  useful 
feedback  can  be  obtained  on  a  continuing  basis.  An  example  of  a  feedback  sys¬ 
tem  which  normally  provides  for  periodic  coordination  between  training  develop¬ 
ers  and  evaluators  is  the  Missile  and  Munitions  Evaluation  conducted  by  the 
Missile  and  Munitions  Center/School  (MAKE,  1982)  . 

Field  Contacts  For  Training  Developers 

Training  developers  develop  their  products  in  the  relative  isolation  of 
the  Center/School.  While  some  training  developers  have  gained  extensive  field 
experience  in  previous  assignments,  their  knowledge  is  rapidly  outdated  by 
changes  in  doctrine,  equipment,  and  the  threat.  Some  training  developers  in¬ 
dicated  during  interviews  that  they  must  constantly  struggle  to  keep  up  with 
changes  in  the  field.  To  provide  products  that  are  responsive  to  the  field's 
needs,  DTD  personnel  must  increase  and  maintain  their  direct  contacts  with 
field  personnel*  Two  approaches  to  this  are  suggested  below. 

Interviews  with  field  personnel  at  the  Center/School.  Perhaps  the  most 
available  and  underutilized  resource  for  obtaining  external  feedback  is  field 
personnel  who  return  to  the  Center/School  for  training  in  advanced  NCO  or  offi¬ 
cer  courses.  With  their  experiences  in  the  field  still  fresh  in  their  minds 
and  the  distractions  of  their  field  duties  temporarily  removed,  they  can  provide 
much  useful  feedback  to  the  Center/School.  These  soldiers  should  be  interviewed 
individually  by  DOES  and  DTD  personnel  shortly  after  their  arrival  at  the 
Center/School.  Structured  interview  formats  should  be  used  in  sessions  requir¬ 
ing  no  more  than  an  hour  of  the  interviewee's  time.  Interviews  could  cover 
practically  any  subject  matter  than  can  be  asked  of  the  same  soldiers  in  the 
field  and  need  not  be  restricted  to  perceptions  of  graduate  performance,  as  is 
presently  done.  In  fact,  it  might  be  instructive  to  ask  them  the  same  questions 
that  are  asked  of  their  counterparts  in  the  field  to  determine  if  the  feedback 
obtained  is  comparable.  If  the  information  obtained  from  soldiers  returning  to 
the  Center/School  varies  little  from  that  obtained  in  the  field,  then  the  number 
of  personnel  interviewed  during  field  visits  could  be  reduced.  Interviews  con¬ 
ducted  with  field  personnel  at  the  Center/School  are  more  convenient  and  less 
expensive  tlmn  the  same  interviews  conducted  in  the  field.  DTD  personnel  might 
also  construct  short  questionnaires  to  answer  specific  questions  of  immediate 
concern  about  their  products  that  are  not  addressed  in  field  interviews  or 
questionnaires.  Administering  these  questionnaires  to  field  personnel  at  the 
institution  would  provide  quick  and  cost-effective  feedback.  Field  personnel 
temporarily  located  at  the  Center/School  are  a  feedback  source  that  should  be 
utilized  more  extensively. 
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Field  visits.  Another  way  in  which  contacts  between  training  developers 
and  field  personnel  could  be  increased  would  be  to  include  training  develop¬ 
ment  personnel  on  the  evaluation  teams  that  routinely  visit  units  in  the  field. 
Although  the  DOES  occasionally  includes  personnel  from  other  directorates/ 
departments  on  its  external  evaluation  team,  the  number  of  times  that  training 
developers  have  been  included  is  far  too  few.  The  DOES  should  include  at  least 
one  training  developer  on  every  field  visit. 

Training  developers  should  accompany  DOES  personnel  as  active  observers. 
Although  they  would  not  be  the  key  data  collectors,  the  training  developers 
might  be  involved  in  administering  questionnaires,  observing  unit  operations, 
and  conducting  interviews  with  lower  ranking  officers,  NCO's,  and  enlisted 
personnel.  It  is  important  for  training  developers  visiting  field  units  to  have 
a  definite  but  flexible  agenda.  Prior  to  the  trip  questionnaires  and  structured 
interviews  should  be  developed  and  plans  made  for  collecting  the  data  and  making 
observations.  Several  training  developers  expressed  concern  during  the  inter¬ 
views  that  they  would  be  allowed  to  see  only  what  units  want  them  to  see  in  the 
field.  Data  collectors  should  be  trained  to  get  around  this  problem  by  looking 
for  objective  indicators  of  unit  performance.  Much  can  be  learned  through  ob¬ 
jective  structured  observation  of  unit  activities;  this  point  is  further  ad¬ 
dressed  in  a  discussion  of  the  validity  of  feedback  below. 

In  addition  to  providing  badly  needed  contact  with  field  personnel,  there 
are  other  advantages  to  including  training  developers  on  field  visits.  Training 
developers  are  likely  to  have  more  confidence  in  feedback  that  they  themselves 
have  been  involved  in  collecting.  They  will  be  more  motivated  to  develop  methods 
for  obtaining  the  kinds  of  feedback  they  need,  since  they  know  that  the  needed 
information  will  be  collected.  They  may  also  come  to  better  appreciate  the 
difficulty  of  obtaining  useful  feedback  from  the  field.  Also,  as  suggested  ear¬ 
lier,  training  developers  might  be  able  to  develop  detailed  follow-up  questions 
during  interviews  in  the  field,  due  to  their  direct  experience  with  the  products 
being  evaluated. 

There  are,  however,  at  least  two  potential  problems  with  sending  training 
developers  to  collect  feedback  in  the  field.  The  first  is  the  additional  cost 
of  sending  additional  people  to  the  field.  However,  this  procedure  might  save 
money  in  the  long  run  if  it  leads  to  improved  training  materials  and  better 
trained  soldiers.  It  might  also  lessen  the  number  of  training  evaluators 
visiting  the  field  and  reduce  the  need  for  training  developers  to  conduct  sep¬ 
arate  visits  on  their  own.  The  second  problem  is  that  training  developers 
might  introduce  bias  into  the  data  they  collect.  They  might  inadvertently,  or 
purposely,  make  their  products  (i.e.,  training  materials  and  trained  soldiers) 
look  better  than  they  actually  are  by  asking  questions  in  a  biased  manner  or 
by  not  being  objective  during  interviews.  For  this  reason  training  developers 
should  receive  instruction  in  objective  methods  of  data  collection  before  they 
visit  the  field,  and  the  data  they  collect  should  be  used  only  internally  for 
identifying  problems  and  correcting  deficiencies  and  not  for  more  general  eval¬ 
uative  purposes. 
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Validity  Of  Feedback 

Subjective  versus  objective  measures.  Much  of  the  feedhack  presently 
available  to  training  developers  is  subjective  in  that  it  is  based  on  the 
perceptions  and  opinions  of  unit  personnel.  Among  the  more  subjective  data 
are  responses  to  student  questionnaires,  ratings  of  graduate  performance,  CODAP 
survey  results,  and  professionalism  survey  results.  If  subjective  measures 
like  these  are  going  to  be  used,  then  their  accuracy  or  validity  must  be  demon¬ 
strated  by  comparing  them  with  more  objective  measures,  such  as  results  of  tests 
and  observations.  If  the  subjective  measures  do  not  compare  favorably  with  the 
objective  ones,  then  the  subjective  measures  must  be  replaced  by  more  accurate 
ones. 


Ratings  and  other  estimates  of  graduate  proficiency  on  specific  tasks,  such 
as  those  collected  by  the  DOES  and  in  CO DAP  surveys, ""have  not  been  generally 
shown  to  be  accurate  when  compared  with  more  objective  performance  measures 
(Burnside,  1982)  .  Therefore,  DOES  personnel  should  compare  their  subjective 
measures  of  task  performance  with  the  results  of  objective  performance  meas¬ 
ures  such  as  hands-on  tests.  Unless  a  significant  relationship  can  be  demon¬ 
strated  between  performance  estimates  and  hands-on  performance,  the  former  meas¬ 
ures  will  have  to  be  abandoned  for  more  accurate  but  expensive  performance 
measures . 

The  accuracy  of  estimates  of  task  criticality  and  frequency  of  performance 
as  collected  by  the  DOES  and  in  CODAP  surveys  has  also  not  been  demonstrated. 
Burnside  (1982)  found  few  studies  supporting  the  accuracy  of  either  frequency 
or  criticality  estimates.  Estimates  of  time  spent  on  specific  tasks  and  fre¬ 
quency  of  performing  these  tasks  are  generally  inflated,  leading  to  the  neces¬ 
sity  to  use  relative  rather  than  absolute  judgments.  To  determine  how  accurate 
subjective  measures  of  frequency  of  task  performance  are,  the  DOES  should  ob¬ 
serve  the  frequency  of  performance  of  a  selected  set  of  tasks  in  a  field  setting 
and  compare  this  objective  measure  of  frequency  with  frequency  estimates.  Esti¬ 
mates  of  task  criticality  are  notoriously  unreliable.  If  task  criticality  esti¬ 
mates  are  to  be  used  at  all,  special  techniques  may  be  required  to  obtain  them. 
When  soldiers  are  asked  to  rate  criticality,  almost  all  tasks  are  rated  at  the 
high  end  of  the  scale.  That  is,  almost  all  tasks  are  considered  critical.  This 
problem  can  be  overcome  by  forcing  soldiers  to  rank  order  the  tasks  from  most  to 
least  critical,  but  this  becomes  quite  difficult  as  the  number  of  tasks  in¬ 
creases.  Burnside  (1982)  discussed  another  method  for  obtaining  criticality 
estimates  whereby  the  criticality  of  each  task  is  compared  with  that  of  each 
and  every  other  task  in  pairs.  Although  this  paired  comparison  technique  has 
been  shown  to  increase  the  reliability  of  criticality  estimates,  it  becomes  im¬ 
practical  when  the  number  of  tasks  is  more  than  six  or  eight.  Because  the  re¬ 
liability  of  criticality  estimates  is  typically  low,  their  use  is  not  recom¬ 
mended  for  feedback  purposes  unless  their  reliability  can  be  improved  by  using 
special  data  collection  techniques. 

To  be  useful,  measures  of  training  effectiveness  must  provide  detailed, 
accurate  information  so  that  deficiencies  in  the  training  can  be  identified  and 
corrected.  Because  their  primary  goal  is  not  to  evaluate  training,  students  may 


not  make  the  detailed  observations  needed  for  evaluating  and  modifying  training. 
Students'  responses  to  questionnaires  about  the  effectiveness  of  training  may 
consist  primarily  of  gross  judgments  regarding  the  training  they  received.  This 
becomes  more  likely  when  the  questionnaires  ask  students  to  rate  the  instruction 
or  instructor  on  each  of  several  dimensions.  When  student  questionnaires  require 
such  ratings,  they  are  subject  to  all  the  problems  associated  with  ratings  men¬ 
tioned  earlier.  In  addition,  responses  to  student  questionnaires  may  be  affected 
disproportionately  by  the  style  and  charisma  of  the  instructor.  For  these  reas¬ 
ons  student  questionnaires  should  not  be  used  as  the  primary  measure  of  training 
effectiveness.  Generally  training  effectiveness  is  more  accurately  measured  by 
sending  carefully  selected,  well-trained  observers  to  objectively  record  their 
observations  of  training  on  structured  observation  forms.  The  DOES  presently 
sends  observers  to  inspect  institutional  training  but  structured  observation 
forms  are  not  always  used,  and  those  forms  that  are  used  depend  too  heavily  on 
subjective  assessment  of  the  training.  Objective  observation  of  training  coup¬ 
led  with  rigorous  end-of-block  tests  is  recommended  as  the  best  way  to  evaluate 
training.  Both  training  developers  and  evaluators  should  apply  this  approach  in 
both  institutional  and  unit  training.  Training  evaluations  should  be  conducted 
in  accordance  with  guidelines  provided  in  ARI  Research  Products  81-15  through 
81-18  (Kristiansen,  1981;  Kristiansen  and  Witmer,  1982a;  Kristiansen  and  Witmer, 
1982b;  Witmer,  1981). 


Most  of  the  items  on  the  professionalism  survey  call  for  subjective  re¬ 
sponses.  Almost  every  item  calls  for  either  an  attitude,  an  opinion,  or  an  eval¬ 
uation  of  some  aspect  of  the  Army  as  a  profession.  The  purpose  of  the  profes¬ 
sionalism  survey  is  to  identify  attitudes  or  opinions  that  may  relate  to  person¬ 
nel  problems  such  as  officer  or  NCO  attrition.  However,  the  DOES  has  made  no 
effort  to  compare  survey  results  with  actual  attrition  rates,  and,  as  best  as 
could  be  determined,  the  validity  of  professionalism  surveys  has  never  been  es¬ 
tablished.  Furthermore,  DOES  personnel  interviewed  indicated  that  these  surveys 
are  of  little  value  in  identifying  personnel  problems.  Unless  the  validity  and 
utility  of  professionalism  or  similar  surveys  can  be  demonstrated,  they  should 
not  be  used  for  feedback  purposes. 


Even  relatively  objective  measures  such  as  tests  and  objective  question¬ 
naires  do  not  always  provide  valid  feedback.  For  example,  tests  may  be  admin¬ 
istered  in  which  the  soldiers  are  given  assistance  or  are  not  tested  to  the  cor¬ 
rect  standards.  On  questionnaires,  the  questions  may  be  worded  such  that  the 
respondent  is  biased  toward  giving  the  answer  that  the  questioner  prefers. 
Questions  asked  during  interviews  may  also  be  worded  so  that  the  interviewee 
gives  the  desired  answer,  and  interviewers  may  influence  the  interviewee  by  the 
way  that  they  react  to  the  answers  provided.  To  increase  the  validity  of  data 
gathered  through  questionnaires  and  interviews,  DOES  and  DTD  personnel  should 
carefully  develop  the  questions  to  be  asked  in  advance,  paying  close  attention 
to  the  wording  of  the  questions.  Questions  should  be  aimed  at  obtaining  factual 
information  rather  than  attitudes  or  opinions,  and  should  only  be  asked  of  those 
soldiers  who  are  likely  to  have  the  relevant  experiences  to  provide  the  informa¬ 
tion  requested.  Questionnaires  and  structured  interviews  developed  for  feed¬ 
back  purposes  must  not  only  provide  information  that  is  valid  but  also  informa¬ 
tion  that  is  useful.  Therefore,  the  questionnaire  or  interview  developer  must 
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take  special  care  to  include  only  those  questions  that  yield  information  having 
a  clearly  identified  use  for  a  specific  directorate/department.  For  additional 
guidance  in  constructing  questionnaires  and  structured  interviews  see  Dyer* 
Josephine,  Wright,  and  Yudowitch  (1976)  or  Kristiansen  and  Witmer  (1982a,  Appen¬ 
dix  B)  . 

Presently,  feedback  to  the  Center/School  is  based  too  much  on  subjective 
measures  and  not  enough  on  objective  ones.  While  interviews  and  questionnaires 
can  provide  valid,  useful  information  when  constructed  properly,  some  of  the 
questions  now  used  by  the  DOES  and  the  lack  of  structure  in  the  interviews  in¬ 
vite  subjective  responses.  Along  with  efforts  to  improve  questionnaires  and 
interviews,  training  developers  and  evaluators  should  increase  the  use  of  more 
objective  methods,  such  as  observation  and  hands-on  testing.  Although  these 
methods  are  relatively  costly  and  time-consuming,  their  application  to  a  small 
sample  of  units  may  very  well  yield  more  useful  feedback  than  would  subjective 
measures  gathered  from  every  unit  in  the  Army. 

Criteria  for  samples.  Maintaining  the  objectivity  of  feedback  does  not 
start  or  end  with  the  collection  of  data.  To  insure  that  the  data  are  represen¬ 
tative  of  the  units  visited  and  field  units  in  general,  the  sample  of  soldiers 
from  whom  the  data  are  obtained  must  be  carefully  selected.  Care  must  be  taken 
to  select  soldiers  of  different  ranks  and  avoid  getting  all  feedback  from  the 
senior  members  of  units.  At  the  present  time,  the  DOES  makes  little  or  no  effort 
to  specify  the  size  or  composition  of  the  sample  to  insure  that  it  is  representa¬ 
tive.  The  DOES's  failure  to  specify  clearly  in  advance  the  sample  from  which 
data  will  be  obtained  not  only  adversely  affects  the  general izability  of  the 
data,  but  also  reduces  objectivity  in  decision-making. 

Several  of  the  DOES  personnel  interviewed  expressed  difficulty  in  deciding 
how  many  times  a  problem  should  be  mentioned  by  field  personnel  before  it  could 
be  considered  serious  enough  to  warrant  taking  some  action.  This  difficulty 
arose  because  the  DOES  has  not  established  decision-making  criteria  for  deciding 
when  to  act.  Without  such  criteria  the  decision  to  include  a  problem  as  feed¬ 
back  to  elements  of  the  Center/School  is  a  subjective  decision.  DTD  personnel 
interviewed  indicated  an  awareness  of  this  problem  and  several  of  them  expressed 
a  hesitancy  to  accept  feedback  from  the  DOES  as  valid,  since  they  did  not  know 
the  extent  of  problems  identified.  Decision-making  criteria  are  essential  to 
maintaining  the  objectivity  of  feedback  and  they  should  be  developed  by  the  DOES. 
But  to  develop  these  criteria,  DOES  personnel  need  to  know  the  size  and  composi¬ 
tion  of  the  sample  upon  which  the  decision  will  be  based,  the  questions  to  be 
asked  of  the  sample,  and  the  kinds  of  responses  to  each  question  that  indicate 
problems  which  can  be  solved  by  the  Center/School .  While  the  use  of  decision¬ 
making  criteria  does  not  guarantee  correct  decisions  will  be  made,  it  does  insure 
that  consistent,  objective  decisions  that  do  not  vary  from  one  decision-maker  to 
another  will  be  made. 

Data  Analysis  And  Management  System 

Neither  the  DOES  nor  the  DTD  addressed  in  this  study  has  an  adequate  system 
for  analyzing,  maintaining,  or  distributing  the  feedback  that  they  get.  Although 
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both  the  DOES  and  the  DTD  have  access  to  computer  facilities  in  other  locations 
at  the  Center/School,  neither  has  a  computer  on-site  that  can  be  used  for  stor¬ 
age  and  analysis  of  feedback  data.  The  lack  of  an  on-site  computer  facility 
tends  to  discourage  thorough  analysis  of  the  data,  since  the  analysis  must 
either  be  performed  by  hand  or  through  a  remote  computer  terminal.  Presently 
DOES  and  DTD  personnel  store  the  feedback  they  receive  in  file  drawers  through¬ 
out  their  offices.  This  method  of  storing  data  quickly  becomes  cumbersome  with 
different  data  in  different  locations  and  old  data  shoved  to  the  back  of  the 
file  where  it  remains  untouched  until  it  is  destroyed  to  make  room  for  new  data. 
As  a  result,  trends  in  feedback  over  time  are  not  established  and  data  which  may 
be  similar  are  rarely  integrated  or  compared.  This  leads  to  a  lack  of  continu¬ 
ity  in  feedback,  as  was  discussed  earlier. 

The  DOES,  as  the  primary  data  collector  and  feedback  provider  at  the 
Center/School,  needs  an  on-site  computer  for  analyzing  and  maintaining  the  data 
collected.  To  obtain  maximum  benefit  from  the  computer,  procedures  must  be  de¬ 
veloped  for  analyzing  and  maintaining  the  data  acquired.  Existing  data  manage¬ 
ment  and  analysis  packages  can  be  used  for  this  purpose  to  a  certain  extent. 

These  packages  may  need  to  be  modified  or  supplemented  by  other  techniques  to 
fit  the  particular  needs  of  the  DOES  or  those  to  whom  the  DOES  provides  feedback. 
Prototype  data  analysis  and  management  techniques  should  be  developed  based  upon 
existing  data  and  expanded  as  more  data  become  available.  These  techniques  may 
include  objective  methods  for  establishing  and  updating  long-term  data  trends, 
procedures  for  displaying  complex  data  in  simple  pictorial  formats,  decision¬ 
making  sub-routines  based  on  objective  criteria  for  automatically  identifying 
critical  problems,  and  methods  to  monitor  suggested  revisions  in  Center/School 
training  in  order  to  determine  their  effects. 

Care  should  be  taken  to  insure  that  the  computer  does  not  become  just  a 
depository  for  large  pools  of  unvalidated  subjective  data.  For  example,  CODAP 
data  are  highly  subjective  and  may  be  out-of-date  by  the  time  they  are  available. 
The  Amy  produces  a  number  of  data  files,  some  of  which  are  based  on  relatively 
objective  information  and  others  that  consist  primarily  of  subjective  data.  The 
DOES  should  maintain  awareness  of  the  methods  used  to  collect  various  types  of 
feedback  data.  Data  obtained  by  objective  methods  such  as  systematic  observa¬ 
tion  or  hands-on  tests  are  usually  reliable  and  can  be  entered  into  the  data 
base  without  further  verification.  Subjective  data,  on  the  other  hand,  should 
always  be  checked  by  comparing  it  to  data  obtained  via  more  objective  methods. 

For  example,  supervisors’  appraisals  of  subordinates'  performance  on  specific 
tasks  are  not  acceptable  for  entry  in  the  data  base  unless  these  appraisals  can 
be  shown  to  strongly  relate  to  measures  of  hands-on  performance.  Similarly 
responses  to  student  questionnaires  may  not  accurately  measure  training  effec¬ 
tiveness,  and  should  not  be  included  in  the  data  base  unless  they  are  shown  to 
relate  to  systematic  observation  or  other  objective  measures  of  training  effec¬ 
tiveness. 

Feedback  data  stored  in  the  computer  should  be  readily  accessible  by  the 
DTD  and  other  training  developers  as  well  as  by  the  DOES.  Ideally  DTD  and  the 
instructional  departments  would  have  easy  access  to  these  data  through  terminals 
located  at  the  DTD  or  the  instructional  departments.  An  appropriate  command 


from  a  terminal  to  the  computer  would  create  a  printout  of  the  various  types  of 
data  (e.g,,  test  results,  responses  to  questionnaires)  available  in  the  data 
base  and  the  options  and  procedures  available  for  analyzing  or  displaying  them. 
From  this  list  the  user  could  select  the  type  of  data  and  method  of  analysis  to 
be  employed.  Subsequent  instructions  could  be  used  to  select  the  time  frame 
and  soldier  population  of  interest.  With  such  a  system  the  user  could  quickly 
retrieve  the  data  needed. 

The  DOES  computer  could  be  used  to  quickly  generate  tables,  graphs,  sta¬ 
tistics,  and  summaries  of  major  problems  observed  within  a  given  time  frame  for 
inclusion  in  monthly  or  quarterly  reports.  This  would  decrease  the  amount  of 
time  required  by  DOES  personnel  to  produce  a  report,  thereby  increasing  the 
time  available  for  collecting  data  or  conducting  special  studies.  These  reports 
could  be  stored  permanently  in  the  computer  and  could  be  readily  accessible  to 
users  through  terminals.  Users  could  also  create  their  own  reports  for  special 
purposes  by  selecting  the  appropriate  analysis  options  and  obtaining  a  printout 
through  a  terminal.  Direct  access  to  feedback  data  and  reports  based  on  these 
data  would  greatly  speed  and  increase  the  flow  of  feedback  from  training  evalu¬ 
ators  to  training  developers.  It  would  also  allow  feedback  to  be  tailored  to 
the  needs  of  users,  resulting  in  closing  of  the  continuous  feedback  loop  dis¬ 
cussed  earlier. 

Summary 

Training  developers  are  not  getting  all  the  feedback  they  need  for  improv¬ 
ing  training  programs  and  materials.  Additional  feedback  is  needed  on  use  of 
training  materials  and  more  specific  feedback  is  needed  on  graduate  performance, 
including  information  identifying  which  tasks  and  parts  of  tasks  soldiers  have 
difficulty  performing  and  the  reasons  for  poor  performance.  In  providing  this 
feedback  it  is  important  for  training  evaluators  to  specify  the  sample  and  time 
frame  from  which  the  data  were  obtained  and  the  methods  used  for  obtaining  the 
data. 


One  problem  with  the  flow  of  feedback  is  the  lack  of  coordination  between 
training  developers  and  evaluators.  Training  developers  need  to  become  more 
involved  in  the  development  of  questions  used  in  data  collection  instruments 
and  in  the  data  collection  process  itself.  Feedback  sources  available  at  the 
Center/School  should  be  utilized  more  extensively,  and  training  developers 
should  accompany  training  evaluators  on  visits  to  field  units.  Training  devel¬ 
opers  and  evaluators  should  interact  continuously  in  order  to  close  the  feed¬ 
back  loop. 

Presently  both  the  DOES  and  the  DTD  are  relying  too  heavily  on  subjective 
data,  such  as  proficiency,  frequency,  and  criticality  estimates.  Subjective 
data  should  not  be  used  unless  their  validity  can  be  established.  More  em¬ 
phasis  should  be  placed  upon  relatively  objective  measures  such  as  observations 
and  hands-on  tests.  Questionnaires  and  interviews  should  be  upgraded  by  care¬ 
fully  wording  each  question  and  weeding  out  questions  that  elicit  unreliable 
subjective  responses.  The  sample  to  whom  the  questionnaires  and  interviews  are 
to  be  administered  must  be  specified  in  advance  and  carefully  selected. 
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Objective  procedures  utilizing  set  criteria  should  be  used  for  interpreting 
data  and  identifying  significant  problems,  rather  than  the  subjective  methods 
presently  used. 

An  automated  data  analysis  and  management  system  is  needed  to  insure  that 
data  collected  for  feedback  purposes  are  thoroughly  analyzed,  maintained  over 
time,  and  efficiently  distributed  to  those  who  need  them.  This  system  should 
be  physically  located  at  the  DOES  facilities,  but  it  should  be  accessible  via 
terminals  to  the  DTD  and  other  training  developers.  With  the  proper  choice 
of  data  analysis  and  management  packages,  an  automated  data  management  system 
could  perform  many  useful  functions,  including  identification  of  trends, 
automated  decision-making,  and  the  efficient  generation  and  distribution  of 
reports.  The  availability  of  valid  data  in  such  a  system  will  go  a  long  way 
toward  making  feedback  the  continuous  interactive  process  that  it  should  be. 
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APPENDIX  A 
INTERVIEW  FORK 
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INFORMATION  FEEDBACK  NEEDS  OF  CENTER/ SCHOOL 
D IRECTORATES /DEPARTMENTS 


Dlr/Dept _ 

Interviewee 


INTERVIEW  WORKSHEET  1 

_  Dlv/Br 

_  Date_ 

INTRODUCTION 


1.  Purpose  of  project 

a.  Design  of  feedback  system  for  Centers /Schools. 

b.  Feedback  on  both  soldier  performance  and  use  of  training  methods. 

c.  Interviewing  divislon/branch  chiefs  In  DTD,  OAFMS,  and  other  agencies  as  needed 

2.  Information  flow  schematic 

a.  Sources 

b.  Dlviaion/branch  processing  of  information 

c.  Feedback  from  users 

3.  Conduct  of  interview 

a.  Organized  by  type  of  information 

b.  Not  an  evaluation  or  audit  of  anyone's  job 

c.  Want  opportunity  to  come  back  later 

INITIAL  INQUIRY 

What  types  of  information  does  your  divislon/branch  currently  get  (collect  or* 
receive >  that  relatee  to  performance  of  soldiers  or  use  of  training  materials  in  the 
field? 


Soldier  Performance 


Use  of  Materials 


INFORMATION  FEEDBACK  NEEDS  OF  CENTER/ SCHOOL 
DIRECTORATES /DEPARTMENTS 

INTERVIEW  WORKSHEET  2 

Dir/Dept _  Div/Br _ 

It— 

1.  Ia  what  fora  do  you  get  this  information?  Ia  It  useful  end  reliable  in  this  fora? 


2.  From  where  do  you  get  it?  Who  in  your  division/branch  gets  it? 


3.  What's  dona  with  the  information  in  your  division/branch?  (Is  its  fora  changed? 
Is  it  analyzed  or  otherwise  processed?  Is  it  filed?  Where?  What  conclusions  and 
recooaendatlons  are  developed?  What  reports  are  generated?) 


4.  Where  are  the  results  of  your  processing  of  the  information  sent?  What  feedback 
do  you  receive  froa  users  of  the  information?  If  aore  feedback  is  needed,  what  kind? 


3.  What  document  or  regulation  prescribes  the  use  of  this  information? 


A- 3 


FT  5478 


Dir/Dept _  Div/Br 

Item _  _ _  _ 


Additional  Comments  (Indicate  whether  comnenta  are  general  or  keyed  to  questiona  on 
previous  page) : 


/ 
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FT  5478 


INFORMATION  FEEDBACK  NEEDS  OF  CENTER/SCHOOL 
DIRECTORATES /DEPARTMENTS 


INTERVIEW  WORKSHEET  3 

Dir/Dept _ _ _  Dlv/Bg 

CLOSING  INQUIRY 

What  other  types  of  feedback  on  soldier  performance  or  use  of  materials  do 
you  need  that  you  don't  currently  get?  In  what  ways  can  feedback  from  the  field 
be  Improved? 


FT  3478 


A- 5 


a. 


appendix  b 

training  inspection  report 

DATA  COLLECTION  FORM 


B-l 


1.  Did  the  trainer  give  a  diagnostic  examination  prior  to  the 
Instruction? 

2.  If  so,  how  many  GOs  _ ,  NO  GOs  _ . 

3.  What  was  the  standard  for  a  GO?  _ _ 

Did  he  diagnostic  exar.-.  matca  the  training  and  the  post  test? 

•j .  Did  the  trainer  have  the  lesson  plan  prepared? 

0 .  Was  the  lesson  plan  IAW  the  soldiers  manual  task,  condition, 
and  standards? 

7.  Are  the  tasks,  conditions,  and  standards  correct? 

8.  Were  the  tasks,  conditions,  and  standards  explained  adequately 
to  the  student? 

g.  Did  the  trainer  give  the  reason  for  the  subject? 

10.  Was  the  "need  to  know"  emphasized? 

11.  Did  the  training  match  the  training  objective? 

12.  Did  the  trainer  display  adequate  knowledge  of  the  subject 
matter? 

13.  Did  the  trainer  give  all  necessary  Information? 

i.l.  Did  the  trainer  present  any  unnecessary  information? 

nixi  *>..•  trainee  ute  understand,  bln  wurds? 

>t .  Did  the  trainer  demonstrate  how  to  perform  the  tasks 
■  rrectly? 


1.  Karo  the  students  afforded  tine  for  practical  exercises? 


1h.  Did  the  students  complete  the  task  correctly,  at  least  once. 
Without  coaching  or  assistance? 


19.  Was  there  enough  tine  for  hands  on  training/practice? 

20.  ;<as  immediate  feedback  given  to  the  students  after  the 
practice? 

21.  Did  the  students  actively  participate? 


22.  Were  the  students  encouraged  to  ask  questions? 


23.  Did  the  trainer  show  interest  in  helping  the  soldier  to 
learn? 


24.  Was  the  training  site  well  organised? 


25.  Did  the  trainer  arrange  the  training  so  all  could  see  and 
hear? 


26.  Were  all  required  support  materials  and  personnel  present 
and  on  tine? 


27.  Was  all  support  equipnent  operational? 


28.  Were  the  training  materials  and  handouts  called  for  used? 
If  not,  why  not? 


29 .  were  training  aids  introduced  and  explained  as  part  of  the 
m:. "ruction?  If  not,  why  not? 


Did  the  students  have  access  to  the  cru.o-aent/trair.ing  aids? 

n ,  why  nc r “ 


.,2 .  What  was  the  ratio  of  instructors/AXs  per  student? 
V .f.r,  it  adequate  for  the  subject? 


B-4 


YES 

NO 

32.  What  was  the  ratio  of  equipment/training  aids  per 
student? 

Was  it  adequate  for  the  subject 

33. 

Were  the  Als  used  effectively? 

34. 

Did  the  Ms  display  adequate  knowledge  of  the  subject  matter? 

35.  Was  faulty  performance  of  either  the  instructor  or  student 
identified  and  corrected? 

36. 

Did  the  soldiers  receive  a  test  on  the  subject? 

37.  How  long  did  the  test  occur  after  the  completion  of 
training? 

38. 

■ 

Was  the  test  as  close  as  possible  to  the  job  environment? 

39. 

Did  the  training  objective  and  the  training  match  the  test? 

40. 

Were  the  pass/fail  standards  clear  and  correct? 

41. 

Were  the  paaa/fail  standards  the  same  for  all  students? 

42. 

Were  the  specified  standards  applied? 

43. 

Were  the  testers  the  same  personnel  as  the  instructors? 

44. 

an  re 

Were  the  students  given  immediate  feedback  on  their  perform- 
after  testing? 

Ki-r  my  1  IrtlliMl'l  il  1  *tl  tyU't  •  >r  ri  i  vM.  i'  ‘ 

1 

4. 

ti  i 

Tj..c 

b 

Ir.-.cr^uptic  tib 

2. 

v 

lighting 

5- 

•rr*i.iir.q  Araa /Facility 

d. 

Temperature 
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4-'‘.  Was  all  student  training  time  used  effectively?  If  no 
why  not? 


YES 


47.  Did  the  trainer  provide  a  summary  of  tasks,  conditions,  and 
standards? 


48.  Was  concurrent  traininq  utilized  during  the  block  of 
instruction? 

49.  If  concurrent  training  was  utilized,  was  it  effective? 

50.  If  test  given: 

Number  Tested  _ 

Number  GOs  _ 

Number  NO  GOs  _ 

Number  Retested  _ 

Number  of  GOs  on  Retest 


REMARKS  SECTION: 


RECOMMENDATIONS : 


EFFECTIVENESS  OF  TRAINING: 

_  Accomplished  the  training  objective 

_  Did  not  accomplish  the  training  objective 

Evaluators  Signatures 
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